classification
Title: setup.py: _ctypes won't get built when system ffi is only in $PREFIX
Type: compile error Stage: resolved
Components: Build, Cross-Build Versions: Python 3.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: How to link with a non-system libffi?
View: 14527
Assigned To: Nosy List: Alex.Willmer, Ivan Levchenko, Paul Hege, doko, mhpruitt, ned.deily, pmpp, rbelio, zach.ware
Priority: normal Keywords:

Created on 2017-10-06 05:14 by pmpp, last changed 2020-05-08 03:21 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
build.log rbelio, 2018-08-02 13:51
config.log rbelio, 2018-08-02 14:49
Messages (12)
msg303799 - (view) Author: pmp-p (pmpp) * Date: 2017-10-06 05:14
--with-system-ffi is mandatory for linux build but no way is provided ( eg --with-ffi-includes= --with-ffi-libs= ) so setup.py can detect libffi already built in $PREFIX and $EPREFIX/lib.

even if cflags/ldflags are ok , _ctypes build will be skipped with reason:
INFO: Could not locate ffi libs and/or headers

test condition: crystax-ndk toward android-19 on ubuntu xenial 32 bits
msg303801 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2017-10-06 06:04
> even if cflags/ldflags are ok 

Might be this. setup.py doesn't use -I flags in $CFLAGS. It looks into $CPPFLAGS only.
msg322948 - (view) Author: Roy Belio (rbelio) Date: 2018-08-02 12:49
Also happens on suse 11 x86_64 with python 3.7
Same issue exactly building locally (but with cross compiling flag for system independency)
Building our own libffi 3.2.1 and adding it in the CPPFLAGS includes.
msg322952 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-08-02 13:19
Hi Roy, mind sharing the complete build log? Either attaching the log to this issue or pasting a link is fine.
msg322955 - (view) Author: Roy Belio (rbelio) Date: 2018-08-02 13:51
Sure, I'll attach it. one more thing to mention is that during configure it printed: configure: WARNING: --with(out)-system-ffi is ignored on this platform

We are also providing all the dependency libraries ourselves (building, linking, h files and what not) so that should tkae into consideration.
on Suse 11 x86 build is working fine.
and we are using gcc 4.3.4
configure arguments passed:
--prefix=/root/rc3/dist --enable-shared --enable-ipv6 --with-dbmliborder=gdbm --with-gcc --with-system-ffi --with-openssl=/root/rc3/dist
msg322961 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-08-02 14:35
A possible case might be that pkg-config on your build machine is not properly configured. Could you check the value of LIBFFI_INCLUDEDIR in config.log? It shuold either be an empty string or /root/rc3/dist/lib/libffi-3.2.1/include.

As a reference, the following command is used to compute LIBFFI_INCLUDEDIR: https://github.com/python/cpython/blob/v3.7.0/configure.ac#L2936
msg322965 - (view) Author: Roy Belio (rbelio) Date: 2018-08-02 14:49
as seen in the config.log:
LIBFFI_INCLUDEDIR='/root/rc3/dist/lib/libffi-3.2.1/include'

I'm attaching the config.log just in case
msg322967 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2018-08-02 15:08
Thanks for providing the info. However, I'm sorry as the case is beyond my knowledge. Here's the relevant section in CPython that handles LIBFFI path searching: https://github.com/python/cpython/blob/v3.7.0/setup.py#L1984-L2002. You may want to examine variables there to see if there's something unusual.
msg323425 - (view) Author: Matt Pruitt (mhpruitt) Date: 2018-08-11 23:10
Also ran into this issue while building Python in an isolated environment.  Realized that libffi is installing into the $EPREFIX/lib64 directory of our build environment.

Despite pkg-config returning the correct directory for linking libffi, it took explicitly setting LDFLAGS to the correct lib64 directory in my build process for the Python build to succeed.

I noticed from your build.log that you don't explicitly add your lib64 to your LDFLAGS.  Perhaps give that a shot?
msg333383 - (view) Author: Ivan Levchenko (Ivan Levchenko) Date: 2019-01-10 14:31
Was having the same issue compiling python 3.7.1 against locally compilied libffi 3.2.1. 
Setting CPPFLAGS and LDFLAGS was not enough and was still getting the same error: INFO: Could not locate ffi libs and/or headers
Everything worked as soon as i added PKG_CONFIG_PATH to point to the location of the directory that had libffi.pc
For me that was /foo/bar/distrib/libffi-3.2.1/x86_64-unknown-linux-gnu 
Can test if it works with this:
pkg-config libffi --cflags-only-I
msg348109 - (view) Author: Paul Hege (Paul Hege) Date: 2019-07-18 10:26
I had also received the same warnings when running make:
"INFO: Could not locate ffi libs and/or headers"
and "Failed to build these modules: _ctypes". Although Python could still be installed, the pip command was not installed, and installing it using get_pip.py just led to an error indicating that the module "_ctypes" was not found.

I can confirm that setting CPPFLAGS and LDFLAGS as well as PKG_CONFIG_PATH, as suggested my Mr. Levchenko, allowed me to build and install python 3.7.1 locally without root privileges with ctypes and pip included and working, using a libffi also compiled from source.
msg368414 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-05-08 03:21
Thanks for the report and for the analysis. There have been a number of reports over the years about problems trying to build and/or execute Pythons with an external copy of libffi. They have become more pressing now that we no longer vendor the source of libffi in Python source releases. To help focus on the issues, we are consolidated the discussion in Issue14527, one of the earliest opened issues.
History
Date User Action Args
2020-05-08 03:21:22ned.deilysetstatus: open -> closed

superseder: How to link with a non-system libffi?

nosy: + ned.deily
messages: + msg368414
resolution: duplicate
stage: resolved
2019-07-18 10:26:14Paul Hegesetnosy: + Paul Hege
messages: + msg348109
2019-01-28 03:43:38yan12125setnosy: - yan12125
2019-01-10 14:31:24Ivan Levchenkosetnosy: + Ivan Levchenko
messages: + msg333383
2018-08-11 23:10:28mhpruittsetnosy: + mhpruitt
messages: + msg323425
2018-08-02 15:08:05yan12125setmessages: + msg322967
2018-08-02 14:49:24rbeliosetfiles: + config.log

messages: + msg322965
2018-08-02 14:35:40yan12125setmessages: + msg322961
2018-08-02 13:51:46rbeliosetfiles: + build.log

messages: + msg322955
2018-08-02 13:28:47ppperrysettitle: setup.py: _ctypes won't getbuilt when system ffi is only in $PREFIX -> setup.py: _ctypes won't get built when system ffi is only in $PREFIX
2018-08-02 13:19:23yan12125setmessages: + msg322952
components: + Build
2018-08-02 12:49:52rbeliosetnosy: + rbelio
messages: + msg322948
2017-10-06 06:04:49yan12125setmessages: + msg303801
2017-10-06 05:37:21zach.waresetnosy: + doko, zach.ware, yan12125
2017-10-06 05:14:44pmppcreate