This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: How to link with a non-system libffi?
Type: enhancement Stage: patch review
Components: Build, ctypes Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Joshua Merchant, munocat, ned.deily, pda, pmpp, raphael.kl, rnash, rosslagerwall, thesamesam, zhao.wang.unsw
Priority: normal Keywords: patch

Created on 2012-04-08 02:25 by pda, last changed 2022-04-11 14:57 by admin.

Pull Requests
URL Status Linked Edit
PR 20451 open rnash, 2020-05-27 14:41
Messages (13)
msg157776 - (view) Author: Paul A. (pda) Date: 2012-04-08 02:25
I trying to build python using an external libffi package I have installed -- is there some trick in directing --with-system-ffi to the path where it's located.  I don't see clues in config.log or anywhere to help.
msg157777 - (view) Author: Ross Lagerwall (rosslagerwall) (Python committer) Date: 2012-04-08 05:58
If it is in a non-standard location, try setting the environment variables:

LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>
msg157795 - (view) Author: Paul A. (pda) Date: 2012-04-08 16:17
On Sun, Apr 08, 2012 at 05:58:29AM +0000, Ross Lagerwall wrote:
> 
> Ross Lagerwall <rosslagerwall@gmail.com> added the comment:
> 
> If it is in a non-standard location, try setting the environment variables:
> 
> LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a nonstandard directory <lib dir>
> CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if you have headers in a nonstandard directory <include dir>

I'm pretty sure I've already been doing that, but I'll verify again soon
when I get the chance.
msg157932 - (view) Author: Paul A. (pda) Date: 2012-04-10 02:52
Yes, I think my libffi setup is okay, but python apparently doesn't (according to the deeper-down log files I didn't initially know about).  The following is a suspicious-looking snippet from build/temp.hp-ux-B.11.31-ia64-2.7/libffi/config.log... I have to question the usefulness of that linker error message.  My immediate thought is that maybe conftstm.o is a 32-bit object file, but I don't see anything earlier in the log to indicate it was even created.

configure:6159: gcc463 -o conftest  -I. -IInclude -I./Include  -D_TERMIOS_INCLUDED  -I/usr/local/lp64/include -mlp64 -L/usr/local/src/Python-2.7.2 -L/usr/local/lp64/lib conftest.c conftstm.o >&5
ld: Mismatched Data ABI. Expected EF_IA_64_ABI64 but found None in file conftstm.o
Fatal error.
msg158004 - (view) Author: Paul A. (pda) Date: 2012-04-11 02:05
While this is no solution by any means, I think it'd be better for the scenario to be a fatal configure error.  After all, if I say --with-system-ffi, it means I really, really want want to use my own libffi.
msg368399 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-08 01:51
As of current Python 3 releases, like 3.8, the python build no longer vendors a copy of libffi for most Unix systems and for those systems the --with-system-ffi configure option is ignored, i.e. configure and setup.py will always try to find an external libffi.  On most systems, if you use are OK with using the system-supplied libffi, building and execution should usually "just work".  But if you want to use another version of libffi, for example, one that you build from source, it can be very non-intuitive of how to successfully do that *especially* if there is already a system libffi installed.  Depending on the system, you may need to set environment build variables like LDFLAGS, CFLAGS or PKG_CONFIG_PATH, and LD_LIBRARY_PATH for the Python build and/or you might need to override some of the libffi build variables to install in the proper locations for your system.  It seems like, as has been suggested in various places, setup.py should at least try to use pkg-config to find libffi library locations as it already does to find libffi include files. But that might not be enough to get a working setup, i.e. you still might need to set LD_LIBRARY_PATH or the equivalent to find libffi. At the very least, there should be some documentation on how to build with a non-system libffi.
msg368413 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-08 03:20
See also earlier discussions in Issue31710, Issue34823, Issue40488, and Issue40535, closed as duplicated of this issue.
msg370088 - (view) Author: Rupert Nash (rnash) * Date: 2020-05-27 14:43
I have just struggled with building CPython with the _ctypes module. Fundamentally, the problem appears to be that configure uses pkgconfig to find the libffi include directory, while setup.py's detect_ctypes only uses the global list of library directories.

I have made an attempt at fixing this by having configure produce the directory containing libffi (`LIBFFI_LIBDIR`) and setup.py use this. However I've hardly any experience with autotools, so I would be very happy to be corrected if this is no use at all.

The PR is https://github.com/python/cpython/pull/20451
msg370580 - (view) Author: Warren Hardy (munocat) Date: 2020-06-01 21:04
I tried Rupert Nash’s changes to Makefile.pre.in, configure, configure.ac and setup.py. 

I can confirm this changes worked for me. 

I had to edit the Makefile and enter the LIBFFI_INCLUDEDIR and LIBFFI_LIBDIR
this might be due to me being a newb about this stuff, and having some configuration setting wrong.
msg376885 - (view) Author: wang zhao (zhao.wang.unsw) Date: 2020-09-14 17:09
We managed to fixed on the server by following and got it sucuessfully compiled:
-./configure  --with-system-ffi  LDFLAGS="-L /home/XXX/installs/libffi-3.3/lib64/" CPPFLAGS="-I /home/XXX/installs/libffi-3.3/include/"
-LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/XXX/installs/libffi-3.3/lib64/
-make
msg381577 - (view) Author: Raphael Krut-Landau (raphael.kl) Date: 2020-11-21 22:11
To install Python 3.9 locally, with ctypes, this worked for me.

# I created a temp directory
cd ~
mkdir tmp
cd tmp

# I downloaded and installed the latest libffi source from Github
wget "https://github.com/libffi/libffi/releases/download/v3.3/libffi-3.3.tar.gz"
tar xvf libffi-3.3.tar.gz
cd libffi-3.3
./configure --prefix=$HOME && make && make install

# I now had some libffi files in ~/lib/pkgconfig and ~/lib64.
# I added these to LD_LIBRARY_PATH, along with ~/lib.

export LD_LIBRARY_PATH="$HOME/lib/pkgconfig:$HOME/lib:$HOME/lib64:$LD_LIBRARY_PATH"

# I downloaded and installed the latest Python from python.org
cd ~/tmp
wget "https://www.python.org/ftp/python/3.9.0/Python-3.9.0.tgz"
cd ~/tmp/Python-3.9.0
./configure --prefix=$HOME --with-system-ffi LDFLAGS="-L $HOME/lib64/" CPPFLAGS="-I $HOME/tmp/libffi-3.3/include/"
make
make test
make install

# I was then able to run an installer that had shown me the error, "No module named '_ctypes'
pip3 install mycli

I hope this can be useful to you.

Raffi
msg381601 - (view) Author: pmp-p (pmpp) * Date: 2020-11-22 08:10
I think the actual PR20451 with its proper use of pkgconfig could close https://bugs.python.org/issue31710 ( fixing at once android and wasm/wasi compilation )
msg389390 - (view) Author: Rupert Nash (rnash) * Date: 2021-03-23 15:11
Due to some major changes in setup.py around libffi, I've had to re-implement my patch - please see Github. I'd appreciate any comments so we can get this merged. Thanks!
History
Date User Action Args
2022-04-11 14:57:28adminsetgithub: 58732
2021-10-29 16:13:24thesamesamsetnosy: + thesamesam
2021-10-20 13:18:07christian.heimessettype: enhancement
versions: + Python 3.10, Python 3.11
2021-03-23 15:11:04rnashsetmessages: + msg389390
2020-11-22 08:10:11pmppsetmessages: + msg381601
2020-11-21 22:11:12raphael.klsetnosy: + raphael.kl

messages: + msg381577
versions: + Python 3.9, - Python 3.8
2020-09-28 17:04:01pmppsetnosy: + pmpp
2020-09-28 15:06:30pmppsetnosy: - pmpp
2020-09-14 17:11:35pmppsetnosy: + pmpp
2020-09-14 17:09:47zhao.wang.unswsetnosy: + zhao.wang.unsw

messages: + msg376885
versions: - Python 3.9
2020-08-14 16:02:50Joshua Merchantsetnosy: + Joshua Merchant
2020-06-01 21:04:44munocatsetnosy: + munocat
messages: + msg370580
2020-05-27 14:43:24rnashsetmessages: + msg370088
2020-05-27 14:41:28rnashsetkeywords: + patch
nosy: + rnash

pull_requests: + pull_request19705
stage: needs patch -> patch review
2020-05-08 03:27:34ned.deilylinkissue40488 superseder
2020-05-08 03:24:54ned.deilylinkissue40535 superseder
2020-05-08 03:22:08ned.deilylinkissue34823 superseder
2020-05-08 03:21:22ned.deilylinkissue31710 superseder
2020-05-08 03:20:02ned.deilysettype: enhancement -> (no value)
messages: + msg368413
stage: needs patch
2020-05-08 01:51:01ned.deilysettitle: How to link with an external libffi? -> How to link with a non-system libffi?
nosy: + ned.deily

messages: + msg368399

versions: + Python 3.8, Python 3.9, - Python 2.7
components: + Build
2012-04-11 02:05:29pdasetmessages: + msg158004
2012-04-10 02:52:32pdasetmessages: + msg157932
components: + ctypes
2012-04-08 16:17:32pdasetmessages: + msg157795
2012-04-08 05:58:29rosslagerwallsetnosy: + rosslagerwall
messages: + msg157777
2012-04-08 02:25:34pdacreate