This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Cannot cross-compile to more featureful but same tune
Type: compile error Stage: needs patch
Components: Cross-Build Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex.Willmer, ned.deily, pmpp, rossburton
Priority: normal Keywords:

Created on 2019-02-26 16:00 by rossburton, last changed 2022-04-11 14:59 by admin.

Messages (8)
msg336688 - (view) Author: Ross Burton (rossburton) * Date: 2019-02-26 16:00
My build machine is a Haswell Intel x86-64. I'm cross-compiling to x86-64, with -mtune=Skylake -avx2.  During make install PYTHON_FOR_BUILD loads modules from the *build* Lib/ which contain instructions my Haswell can't execute:

| _PYTHON_PROJECT_BASE=/data/poky-tmp/master/work/corei7-64-poky-linux/python3/3.7.2-r0/build _PYTHON_HOST_PLATFORM=linux-x86_64 PYTHONPATH=../Python-3.7.2/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_m_linux_x86_64-linux-gnu python3.7 -v -S -m sysconfig --generate-posix-vars ;\
Illegal instruction
msg336698 - (view) Author: Ross Burton (rossburton) * Date: 2019-02-26 17:01
From what I can tell: sets PYTHON_FOR_BUILD like this if cross-compiling:

PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib _PYTHON_SYSCONFIGDATA_NAME=_sysconfigdata_$(ABIFLAGS)_$(MACHDEP)_$(MULTIARCH) '$interp

Note how PYTHONPATH is set to the source and build paths for Lib/.

The intention appears to be that the in the build is used.  In my case that directory is also full of shared libraries that Python happily loads, and then fails.
msg336727 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-02-27 03:02
Thanks for the report. Unfortunately, as far as I know, we don't claim anywhere to support cross-compiling in general.  As you've probably noticed, the Python Makefile-based build system (for non-Windows platforms) is by no means a conventional GNU autotools one. Over the years there have been some efforts to add support for a handful of specific cross-compilation scenarios, for example compiling for Android.  But I'm afraid you are more or less on your own for the case here.  That said, if you or someone else is motivated to supply some fixes to make more cross-compiling cases work, feel free to reopen the issue and submit the fixes as PRs and they may be considered.  What would probably be the best would be to try to reimplement our build system from the ground up with modern Autotools or something similar but that's a *really* big deal.
msg336998 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-02 09:47

> The intention appears to be that the in the build is used.  In my case that directory is also full of shared libraries that Python happily loads, and then fails.

No, the intention is that the native python finds the sysconfigdata module named $_PYTHON_SYSCONFIGDATA_NAME that is located in the directory where the extensions modules (.so libraries) have been cross-compiled. $_PYTHON_SYSCONFIGDATA_NAME is a pure python module, so it can be loaded by the native python.

Please note that in your first post PYTHONPATH is missing the first part, i.e. the path to this directory obtained with:
    $(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)

When PYTHON_FOR_BUILD runs, it does not load any extension module so the Illegal instruction may be caused by some other problem.
msg337006 - (view) Author: Ross Burton (rossburton) * Date: 2019-03-02 12:17
strace disagrees.  By putting strace in PYTHON_FOR_BUILD and then invoking make sharedmods:

| openat(AT_FDCWD, "/data/poky-tmp/master/work/corei7-64-poky-linux/python3/3.7.2-r0/build/build/lib.linux-x86_64-3.7/", O_RDONLY|O_CLOEXEC) = 3
| read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\20\0\0\0\0\0\0"..., 832) = 832
| fstat(3, {st_mode=S_IFREG|0755, st_size=62104, ...}) = 0
| mmap(NULL, 23880, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f93d7cb9000
| mmap(0x7f93d7cba000, 4096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7f93d7cba000
| mmap(0x7f93d7cbb000, 4096, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f93d7cbb000
| mmap(0x7f93d7cbc000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f93d7cbc000
| close(3)                                = 0
| mprotect(0x7f93d7cbc000, 4096, PROT_READ) = 0
| --- SIGILL {si_signo=SIGILL, si_code=ILL_ILLOPN, si_addr=0x7f93d7cbab10} ---
| +++ killed by SIGILL (core dumped) +++
| Illegal instruction
| Makefile:625: recipe for target 'sharedmods' failed

We do have patches but I don't think they affect this part of the build.
msg337015 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-02 17:05
The cross-compilation rely on the fact that the cross-compiled shared libraries names are constructed using the configure `--host=HOST-TYPE' command line parameter and therefore cannot be loaded by the native compiler that uses a different suffix for those names: in a cross compilation the 'build' machine is identified with a different name than the 'host' machine or it is not a cross-compilation.

Maybe there is a glitch in this assumption ?
msg337029 - (view) Author: Ross Burton (rossburton) * Date: 2019-03-03 12:29
That's exactly the glitch.  I'm cross-compiling to a more powerful IA process from IA.  This *is* a cross-compilation but the triple is the same.  Assuming that you can rely on the loader to not open target binaries when they're on the path to load from is unwise.
msg337034 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2019-03-03 16:16
Re-opening and setting the stage as 'needs patch': at least the cross-compilation should fail with a clear message when the cross-built libraries names and those of the native interpreter have the same suffix.

> Assuming that you can rely on the loader to not open target binaries when they're on the path to load from is unwise.

Meanwhile a workaround may be to build the native interpreter with Py_DEBUG defined (using the --with-pydebug configure option) so that libraries names become different.
Date User Action Args
2022-04-11 14:59:11adminsetgithub: 80306
2020-09-29 12:37:02pmppsetnosy: + pmpp
2019-12-10 08:05:39xdegayesetnosy: - xdegaye
2019-03-03 16:16:15xdegayesetstatus: closed -> open

type: compile error
components: + Cross-Build, - Build

nosy: + Alex.Willmer
messages: + msg337034
resolution: not a bug ->
stage: resolved -> needs patch
2019-03-03 12:29:06rossburtonsetmessages: + msg337029
2019-03-02 17:05:21xdegayesetmessages: + msg337015
2019-03-02 12:17:53rossburtonsetmessages: + msg337006
2019-03-02 09:47:43xdegayesetnosy: + xdegaye
messages: + msg336998
2019-02-27 03:02:47ned.deilysetstatus: open -> closed

nosy: + ned.deily
messages: + msg336727

resolution: not a bug
stage: resolved
2019-02-26 17:01:58rossburtonsetmessages: + msg336698
2019-02-26 16:00:10rossburtoncreate