Title: byte-compile fails for cross-builds
Type: compile error Stage: patch review
Components: Cross-Build Versions: Python 3.6, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: 23968 Superseder:
Assigned To: Nosy List: Benedikt.Morbach, doko, ncoghlan, rpetrov, thomas-petazzoni, xdegaye
Priority: normal Keywords: patch

Created on 2014-10-24 23:52 by Benedikt.Morbach, last changed 2016-09-13 16:36 by xdegaye.

File name Uploaded Description Edit
python_for_build_flag.patch xdegaye, 2016-06-27 08:22 review
py_build_environ.patch xdegaye, 2016-07-01 15:15 review
py_build_environ_2.patch xdegaye, 2016-07-30 16:09 rebased to current tip review
Messages (8)
msg229969 - (view) Author: Benedikt Morbach (Benedikt.Morbach) * Date: 2014-10-24 23:52
For cross-builds the byte-compiling and 2to3 Grammar generation fail.

This is because PYTHONPATH/sys.path includes 'build/lib.$PLATFORM', in which the compiled .so modules reside.
The host python obviously barfs on those, as they are compiled for a different architecture.

However, this directory also contains, which is needed at all times.

A possible fix would be to separate those two.
msg229970 - (view) Author: Benedikt Morbach (Benedikt.Morbach) * Date: 2014-10-25 00:00
alternatively, introduce a $(PYTHON_FOR_INSTALL), which doesn't set PYTHONPATH.

Then run those steps using that, with the Makefile setting PYTHONPATH=$(DESTDIR)$(LIBDEST), which doesn't contain the compiled modules at that point.

The first solution looks cleaner to me, but might be more tricky to implement, as e.g. Modules/getpath.c makes some assumptions about the location of the build dir.
msg267350 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-06-04 23:59
Issue 23968 moves to an arch dependendent directory
msg269359 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-06-27 08:22
The byte-compiling for cross-builds fails because PYTHON_FOR_BUILD defined in configure, sets PYTHONPATH with the directory of the newly built extension modules shared libraries and because PYTHON_FOR_BUILD is used to run in the Makefile. PYTHON_FOR_BUILD, i.e. the native python running on the build system, attempts to load the newly compiled libraries for the foreign host system and fails.

The problem is more difficult to reproduce after changeset 11c4f3f936e7 in issue 22980 since the shared libraries names include now a PLATFORM_TRIPLET that prevents the wrong library to be loaded in most cases. On Android cross-builds, it should be possible to reproduce the failure when the build system and the host system have the same PLATFORM_TRIPLET. In all cases, the interpreter used to run imports the wrong _sysconfigdata module.

To fix this problem, this patch adds the -E option to PYTHON_FOR_BUILD when running compileall in the Makefile and 'cross_compiling' is set. The patch does not fix the fact that, when 'cross_compiling' is not set, PYTHONPATH is set to '$(DESTDIR)$(LIBDEST)' in the Makefile at the beginning of the statement that runs while PYTHON_FOR_BUILD is set to './$(BUILDPYTHON) -E', something is obviously wrong here. So the patch would be simpler if '-E' was used in both cases, but I don't know what would be lost by removing this setting of PYTHONPATH at the beginning of those compileall statements, IOW what was their initial purpose.
msg269617 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-06-30 18:47
> On Android cross-builds, it should be possible to reproduce the failure when the build system and the host system have the same PLATFORM_TRIPLET

I confirm that this is the case. The following backtrace is produced when cross-building and installing Android x86_64 on a linux x86_64:

PYTHONPATH=/sdcard/org.bitbucket.pyona/lib/python3.6   \
	_PYTHON_PROJECT_BASE=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64 _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu python3.6 -Wi /sdcard/org.bitbucket.pyona/lib/python3.6/ \
	-d /sdcard/org.bitbucket.pyona/lib/python3.6 -f \
	-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
Traceback (most recent call last):
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/", line 17, in <module>
    import struct
  File "/sdcard/org.bitbucket.pyona/lib/python3.6/", line 13, in <module>
    from _struct import *
ImportError: cannot open shared object file: No such file or directory
make[3]: [Makefile:1270: libinstall] Error 1 (ignored)

'python3.6' is the native interpreter built from the same source tree as the source tree used for the cross-build. PATH has been set so that this native interpreter is run. Running this same command interactively with the native interpreter gives the same result:

[xavier@bilboquet python-native]$   PYTHONPATH=/home/xavier/src/android/pyona/build/python3.6-android-21-x86_64/build/lib.linux-x86_64-3.6:/home/xavier/src/packages/android/cpython/Lib:/home/xavier/src/packages/android/cpython/Lib/plat-x86_64-linux-gnu ./python3.6
Python 3.6.0a2+ (default:64ed56fbc5e7+, Jun 27 2016, 10:36:45) 
[GCC 6.1.1 20160602] on linux
Type "help", "copyright", "credits" or "license" for more information.
Traceback (most recent call last):
  File "/home/xavier/config/.pystartup", line 14, in <module>
    import readline
ImportError: cannot open shared object file: No such file or directory

Cross-building and installing Android i686 on a linux x86_64 with the same Makefile does not cause the problem.

Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install.
msg269675 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-01 15:15
Adding -E to PYTHON_FOR_BUILD fixes the x86_64 install because in my Android cross-build setup, the native interpreter is built from the same source tree as the cross-build. However, currently the interpreter used for the cross-build may be any interpreter found in PATH and can import an incorrect importlib.util.MAGIC_NUMBER if PYTHONPATH is not set properly. So this new patch does not use -E and splits PYTHON_FOR_BUILD into two variables, PY_BUILD_ENVIRON that sets the environment variables and PYTHON_FOR_BUILD, the interpreter. PY_BUILD_ENVIRON is not used when running compileall.

Tested by running the test suite on the Android i686 and x86_64 emulators (excluding test_ctypes that segfaults python on x86_64). The tests cannot be run with a cross-build where PYTHON_FOR_BUILD is python 3.5.1 because of issue 27434.
msg271698 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-07-30 16:09
Patch rebased to current tip.

To summarize:
* The root cause of the problem reported by Benedikt still exists.
* The problem is hidden now that the shared libraries names include the PLATFORM_TRIPLET.
* But byte-compilation still fails for cross-builds when the 'host' and 'build' PLATFORM_TRIPLET are the same and this occurs for the x86_64 Android build on a x86_64 'build' system.

Matthias, do you have any comment on this patch?
msg276300 - (view) Author: Xavier de Gaye (xdegaye) * (Python committer) Date: 2016-09-13 16:36
Cross-compilation of Android x86_64 on a linux x86_64 host followed by 'make install' does not fail anymore after the changes made in issue 27917. As expected since the triplets are different now when the target is an Android platform.
Date User Action Args
2017-01-05 12:55:23xdegayelinkissue26865 dependencies
2016-09-13 16:39:03xdegayeunlinkissue26865 dependencies
2016-09-13 16:36:21xdegayesetmessages: + msg276300
2016-07-30 16:09:47xdegayesetfiles: + py_build_environ_2.patch

messages: + msg271698
2016-07-29 10:27:13xdegayesetnosy: + thomas-petazzoni
2016-07-01 15:15:21xdegayesetfiles: + py_build_environ.patch

messages: + msg269675
2016-06-30 18:47:38xdegayesetmessages: + msg269617
2016-06-27 08:22:57xdegayesetfiles: + python_for_build_flag.patch

versions: - Python 3.3, Python 3.4
keywords: + patch
nosy: + xdegaye

messages: + msg269359
stage: patch review
2016-06-26 16:32:52xdegayelinkissue26865 dependencies
2016-06-04 23:59:43ncoghlansetnosy: + ncoghlan
dependencies: + rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)
messages: + msg267350
2014-10-26 11:54:16skrahsetnosy: + doko
2014-10-26 09:00:07rpetrovsetnosy: + rpetrov
2014-10-25 00:00:53Benedikt.Morbachsetmessages: + msg229970
2014-10-24 23:53:07Benedikt.Morbachsetversions: + Python 3.5, Python 3.6, - Python 3.2
2014-10-24 23:52:39Benedikt.Morbachcreate