(Tested with Python 3.7, but AFAICT, the situation hasn’t changed in Python 3.8.)

Python’s configure tries to fill the LIBFFI_INCLUDEDIR system variable with the value of -I as returned by pkg-config libffi --cflags-only-I.

Python’s setup.py tries to determine whether libffi is present by searching a couple of well-known library directories.

Both of these checks fail in my environment (think Linux From Scratch). I’m setting the following environment variables:

CPATH=/home/michael/libffi-3.2.1/include
LIBRARY_PATH=/home/michael/libffi-3.2.1/lib64

gcc can build against libffi just fine:

% echo echo '#include <ffi.h>                  
> int main() {}' > foo.c
% gcc -o foo foo.c -lffi
% ldd foo
	linux-vdso.so.1 (0x00007ffe4450a000)
	libffi.so.6 => /home/michael/libffi-3.2.1/lib64/libffi.so.6 (0x00007f3935aba000)
	libc.so.6 => /home/michael/glibc-2.27/lib/libc.so.6 (0x00007f3935902000)
	/lib64/ld-linux-x86-64.so.2 => /home/michael/glibc-2.27/lib/ld-linux-x86-64.so.2 (0x00007f3935aca000)

However, when compiling Python, I’m getting the following error message, resulting in the _ctypes module not being built:

INFO: Could not locate ffi libs and/or headers

Now, one issue is that pkg-config understands CPATH, so calling pkg-config --cflags-only-I is not sufficient to obtain the include directory, one also needs to clear CPATH:

% pkg-config libffi --cflags-only-I      

% CPATH= pkg-config libffi --cflags-only-I
-I/home/michael/libffi-3.2.1/include

After patching this in configure, LIBFFI_INCLUDEDIR is set correctly, but the build still fails, as the libffi shared object is located in my non-standard path, which setup.py doesn’t check.

Without knowing much about the Python internals, it seems to me that both of these issues would be fixed if Python stopped trying to find the libffi include/lib locations and just used pkg-config to detect the required CFLAGS and LDFLAGS, just like Python currently does with openssl.

Is there any reason we can’t use pkg-config for libffi like that?

Thanks!

I have the exact same issue, trying to compile 3.7.1 with a custom libffi location. Note that I must build libffi from source and can't install binaries provided by my distro, I believe this is the origin of the problem. Probably the python build system checks for libffi in some "standard" locations and it doesn't seem possible to use libffi from a custom location.

This is where libffi gets installed after passing --prefix=$HOME/opt to ./configure:


$HOME/opt/lib64/libffi.so.6.0.4
$HOME/opt/lib64/libffi.a
$HOME/opt/lib64/libffi.la
$HOME/opt/lib64/libffi.so.6
$HOME/opt/lib64/libffi.so
$HOME/opt/lib/pkgconfig/libffi.pc
$HOME/opt/lib/libffi-3.2.1/include/ffi.h
$HOME/opt/lib/libffi-3.2.1/include/ffitarget.h
$HOME/opt/share/info/libffi.info

In any case, just to be sure, I've copied the header files to

$HOME/opt/include/ffi.h
$HOME/opt/include/ffitarget.h

And pkg-config works:

[fetch@fetch opt]$ pkg-config --libs libffi
-L/home/fetch/opt/lib/../lib64 -lffi

[fetch@fetch opt]$ pkg-config --cflags libffi
-I/home/fetch/opt/lib/libffi-3.2.1/include

These environment variables are also set:

LD_LIBRARY_PATH=/home/fetch/opt/lib:/home/fetch/opt/lib64

C_INCLUDE_PATH=/home/fetch/opt/include

And still _ctypes fails to build (but python itself (minus _ctypes) compiles successful and works perfectly well).

It would be really great if this could be sorted out because at the moment this bug prevents me from using numpy/scipy with python 3.7.1 (they need _ctypes).

It seems there is a way to fix this:

https://mail.python.org/pipermail/python-list/2018-December/738568.html

LDFLAGS=`pkg-config --libs-only-L libffi` ./configure

Would be nice to document this or make the build system find the libraries and headers in a simpler way.

Thanks for the report and for the analysis. There have been a number of reports over the years about problems trying to build and/or execute Pythons with an external copy of libffi. They have become more pressing now that we no longer vendor the source of libffi in Python source releases. To help focus on the issues, we are consolidated the discussion in Issue14527, one of the earliest opened issues.