I trying to build python using an external libffi package I have installed -- is there some trick in directing --with-system-ffi to the path where it's located.  I don't see clues in config.log or anywhere to help.

If it is in a non-standard location, try setting the environment variables:

LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>

On Sun, Apr 08, 2012 at 05:58:29AM +0000, Ross Lagerwall wrote:
> 
> Ross Lagerwall <rosslagerwall@gmail.com> added the comment:
> 
> If it is in a non-standard location, try setting the environment variables:
> 
> LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
> CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>

I'm pretty sure I've already been doing that, but I'll verify again soon
when I get the chance.

Yes, I think my libffi setup is okay, but python apparently doesn't (according to the deeper-down log files I didn't initially know about).  The following is a suspicious-looking snippet from build/temp.hp-ux-B.11.31-ia64-2.7/libffi/config.log... I have to question the usefulness of that linker error message.  My immediate thought is that maybe conftstm.o is a 32-bit object file, but I don't see anything earlier in the log to indicate it was even created.

configure:6159: gcc463 -o conftest  -I. -IInclude -I./Include  -D_TERMIOS_INCLUDED  -I/usr/local/lp64/include -mlp64 -L/usr/local/src/Python-2.7.2 -L/usr/local/lp64/lib conftest.c conftstm.o >&5
ld: Mismatched Data ABI. Expected EF_IA_64_ABI64 but found None in file conftstm.o
Fatal error.

While this is no solution by any means, I think it'd be better for the scenario to be a fatal configure error.  After all, if I say --with-system-ffi, it means I really, really want want to use my own libffi.

As of current Python 3 releases, like 3.8, the python build no longer vendors a copy of libffi for most Unix systems and for those systems the --with-system-ffi configure option is ignored, i.e. configure and setup.py will always try to find an external libffi.  On most systems, if you use are OK with using the system-supplied libffi, building and execution should usually "just work".  But if you want to use another version of libffi, for example, one that you build from source, it can be very non-intuitive of how to successfully do that *especially* if there is already a system libffi installed.  Depending on the system, you may need to set environment build variables like LDFLAGS, CFLAGS or PKG_CONFIG_PATH, and LD_LIBRARY_PATH for the Python build and/or you might need to override some of the libffi build variables to install in the proper locations for your system.  It seems like, as has been suggested in various places, setup.py should at least try to use pkg-config to find libffi library locations as it already does to find libffi include files. But that might not be enough to get a working setup, i.e. you still might need to set LD_LIBRARY_PATH or the equivalent to find libffi. At the very least, there should be some documentation on how to build with a non-system libffi.

See also earlier discussions in Issue31710, Issue34823, Issue40488, and Issue40535, closed as duplicated of this issue.

I have just struggled with building CPython with the _ctypes module. Fundamentally, the problem appears to be that configure uses pkgconfig to find the libffi include directory, while setup.py's detect_ctypes only uses the global list of library directories.

I have made an attempt at fixing this by having configure produce the directory containing libffi (`LIBFFI_LIBDIR`) and setup.py use this. However I've hardly any experience with autotools, so I would be very happy to be corrected if this is no use at all.

The PR is https://github.com/python/cpython/pull/20451

I tried Rupert Nash’s changes to Makefile.pre.in, configure, configure.ac and setup.py. 

I can confirm this changes worked for me. 

I had to edit the Makefile and enter the LIBFFI_INCLUDEDIR and LIBFFI_LIBDIR
this might be due to me being a newb about this stuff, and having some configuration setting wrong.

We managed to fixed on the server by following and got it sucuessfully compiled:
-./configure  --with-system-ffi  LDFLAGS="-L /home/XXX/installs/libffi-3.3/lib64/" CPPFLAGS="-I /home/XXX/installs/libffi-3.3/include/"
-LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/XXX/installs/libffi-3.3/lib64/
-make

To install Python 3.9 locally, with ctypes, this worked for me.

# I created a temp directory
cd ~
mkdir tmp
cd tmp

# I downloaded and installed the latest libffi source from Github
wget "https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz"
tar xvf libffi-3.3.tar.gz
cd libffi-3.3
./configure --prefix=$HOME && make && make install

# I now had some libffi files in ~/lib/pkgconfig and ~/lib64.
# I added these to LD_LIBRARY_PATH, along with ~/lib.

export LD_LIBRARY_PATH="$HOME/lib/pkgconfig:$HOME/lib:$HOME/lib64:$LD_LIBRARY_PATH"

# I downloaded and installed the latest Python from python.org
cd ~/tmp
wget "https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tgz"
cd ~/tmp/Python-3.9.0
./configure --prefix=$HOME --with-system-ffi LDFLAGS="-L $HOME/lib64/" CPPFLAGS="-I $HOME/tmp/libffi-3.3/include/"
make
make test
make install

# I was then able to run an installer that had shown me the error, "No module named '_ctypes'
pip3 install mycli

I hope this can be useful to you.

Raffi

I think the actual PR20451 with its proper use of pkgconfig could close https://bugs.python.org/issue31710 ( fixing at once android and wasm/wasi compilation )

Due to some major changes in setup.py around libffi, I've had to re-implement my patch - please see Github. I'd appreciate any comments so we can get this merged. Thanks!