classification
Title: setup.py uses build Python's configuration when cross-compiling
Type: behavior Stage:
Components: Cross-Build Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, Rouslan Korneychuk, doko, xdegaye, yan12125, zach.ware
Priority: normal Keywords: patch

Created on 2016-09-24 21:57 by Rouslan Korneychuk, last changed 2016-09-25 09:23 by Rouslan Korneychuk. This issue is now closed.

Files
File name Uploaded Description Edit
setup-fix.patch Rouslan Korneychuk, 2016-09-24 21:57 proposed fix
setup-fix.patch Rouslan Korneychuk, 2016-09-25 02:05 3.6b1 fix
Messages (7)
msg277340 - (view) Author: Rouslan Korneychuk (Rouslan Korneychuk) * Date: 2016-09-24 21:57
When building Python, the setup.py script will use values from sysconfig, even when cross compiling. When cross compiling, the interpreter that runs setup.py is configured for the build system, not the host system, so the wrong values are used. This patch should fix that.

It should be noted that the updated script assumes it is run from the build directory (like the Makefile) and that the script is at the top-level source directory. Also it uses a private method from sysconfig. This is unavoidable without modifying the sysconfig interface (or duplicating code).
msg277343 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-09-25 00:26
Could you try 3.6b1 and see if it works correctly for you? Cross-building support has changed significantly in 3.6.
msg277344 - (view) Author: Rouslan Korneychuk (Rouslan Korneychuk) * Date: 2016-09-25 02:05
Here is an updated patch for 3.6b1. I was able to compile Python with the changes, natively (x86_64 linux) and for ARM, using an Android "isolated toolchain." However, with the ARM build, a file named _sysconfigdata_m_linux_x86_64-linux-gnu.py ended up in build/lib.linux-x86_64-3.6 instead of build/lib.linux-arm-3.6 like the ".so"s did. I don't know what created the file or where it's used. I'm sure I could figure it out if I investigate, but I'll wait for a reply first, since this is new behaviour and I figure someone on here could save me some trouble.
msg277346 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-25 06:29
This is fixed two weeks ago. See issue28046. The relevant fix is at http://bugs.python.org/issue28046#msg275845. Xavier pushed it to default before the 3.6 branch is created. I guess it should already be in 3.6b1??

Current hg tip works for me:

$ find build/21-aarch64-linux-android-4.9/lib/python3.7 -name '*sysconfigdata*.py'
build/21-aarch64-linux-android-4.9/lib/python3.7/_sysconfigdata_m_linux_aarch64-linux-android.py
build/21-aarch64-linux-android-4.9/lib/python3.7/lib-dynload/_sysconfigdata_m_linux_aarch64-linux-android.py

I'm using my own build script: https://github.com/yan12125/python3-android. If different build scripts lead to different result, CPython should handle it. Rouslan, would you like to share your build script?
msg277348 - (view) Author: Rouslan Korneychuk (Rouslan Korneychuk) * Date: 2016-09-25 08:07
I just tried compiling 3.6 unmodified for Android and it failed when trying to build libffi. The error message included these two lines:

configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.

so 3.6b1 is still missing something.


As for sharing my build script, I'm not sure what you are referring to. I just ran:

TOOLKIT_BASE=/home/rouslan/bin/android21-arm-toolchain
TOOLKIT_BIN=$TOOLKIT_BASE/bin/arm-linux-androideabi
~/build/Python-test-3.6.0b1/configure \
    --build=x86_64-unknown-linux-gnu --host=arm-linux \
    --enable-ipv6 \
    CC="$TOOLKIT_BIN-gcc --sysroot=$TOOLKIT_BASE/sysroot -pthread" \
    CXX="$TOOLKIT_BIN-g++ --sysroot=$TOOLKIT_BASE/sysroot -pthread" \
    AR=$TOOLKIT_BIN-ar \
    RANLIB=$TOOLKIT_BIN-ranlib \
    READELF=$TOOLKIT_BIN-readelf \
    CFLAGS="-fPIC -mthumb -march=armv7-a -mfloat-abi=softfp -mfpu=neon -DNDEBUG" \
    LDFLAGS="-fPIC -march=armv7-a -Wl,--fix-cortex-a8" \
    PYTHON_FOR_BUILD=~/build/python-test/python \
    ac_cv_file__dev_ptmx=yes ac_cv_file__dev_ptc=yes ac_cv_little_endian_double=yes
make

I don't know if all of that is even necessary. This is what was needed to build 3.5.0 (version 3.5.0 also required a number of additional changes to compile). There is also a build script I use for my project that incorporates Python, which has additional changes, such as the ability to run from an apk (android package) file, but it's a SCons file and does a whole lot more than just build Python (which it does by running configure and make, anyway).
msg277349 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2016-09-25 08:53
The problem is: setting PYTHON_FOR_BUILD manually breaks cross-compiling. Please remove it, add python3.6 to $PATH and cross-compiling should be OK.

For the problem of libffi: since issue27976 using bundled libffi is deprecated. Please build libffi outside CPython and then use --with-system-ffi.

Some other minor points:
* Android support is still highly experimental. Please consider using the latest mercurial changeset, or at least 3.6.x. 3.5.x requires too many patches.
* Android NDK has deprecated GCC. Mostly CPython is tested on clang.
msg277351 - (view) Author: Rouslan Korneychuk (Rouslan Korneychuk) * Date: 2016-09-25 09:23
> The problem is: setting PYTHON_FOR_BUILD manually breaks cross-compiling. Please remove it, add python3.6 to $PATH and cross-compiling should be OK.

I see.


> For the problem of libffi: since issue27976 using bundled libffi is deprecated. Please build libffi outside CPython and then use --with-system-ffi.

Will do.


> Some other minor points:
* Android support is still highly experimental. Please consider using the latest mercurial changeset, or at least 3.6.x. 3.5.x requires too many patches.

I've had 3.5 running on my Android tablet for a year now, so it's a little late to worry about how many patches it requires (I didn't submit a patch then, because until recently, my changes made it compile *only* for Android).


> * Android NDK has deprecated GCC.

I know :(  Updating my SCons script to use Clang is on my to-do list.
History
Date User Action Args
2016-09-25 09:23:49Rouslan Korneychuksetstatus: open -> closed
resolution: duplicate
messages: + msg277351
2016-09-25 08:53:56yan12125setmessages: + msg277349
2016-09-25 08:07:49Rouslan Korneychuksetmessages: + msg277348
2016-09-25 06:29:53yan12125setnosy: + yan12125
messages: + msg277346
2016-09-25 02:05:18Rouslan Korneychuksetfiles: + setup-fix.patch

messages: + msg277344
versions: + Python 3.6
2016-09-25 00:26:47zach.waresetnosy: + xdegaye, doko, zach.ware
messages: + msg277343
2016-09-24 21:57:46Rouslan Korneychukcreate