classification
Title: Cross compiling on Linux is untested, undocumented, and broken
Type: Stage: patch review
Components: Build, Cross-Build, Documentation, Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: 45886 45909 Superseder:
Assigned To: docs@python Nosy List: Alex.Willmer, brett.cannon, christian.heimes, docs@python
Priority: normal Keywords: patch

Created on 2021-11-23 16:37 by christian.heimes, last changed 2021-11-29 16:23 by christian.heimes.

Pull Requests
URL Status Linked Edit
PR 29752 merged christian.heimes, 2021-11-24 16:18
PR 29753 merged christian.heimes, 2021-11-24 17:57
PR 29754 merged christian.heimes, 2021-11-24 18:01
PR 29835 merged christian.heimes, 2021-11-29 11:45
Messages (7)
msg406853 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-23 16:37
Cross compiling is an approach to compile a program for a different CPU architecture and platform, e.g. compile for an ARM64 (aarch64) or WASM on a x86_64 build system.

Python configure script, Makefile, and setup.py have multiple references to cross compiling. However I could not find any documentation in the devguide or Python docs how to cross compile. We also lack CI (buildbot) to test cross compiling. This lack of awareness and testing leads to breakage of the feature. For example the design of Programs/_freeze_module in main (3.11-dev) is incompatible with cross compiling.

I kinda got cross compiling working with 3.10, but only with some additional hacks and patches. I also ran into other problems like _PYTHON_HOST_PLATFORM env var is not automatically forwarded to setup.py. The helper functions add_multiarch_paths() and add_cross_compiling_paths() break builds for me, too. Cross compiling only works when the methods are commented out.
msg406860 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-23 17:00
I have uploaded a reproducer at https://github.com/tiran/cpython_builddep/compare/cross-aarch64

$ podman run -ti --rm -v $(pwd):/cpython:Z quay.io/tiran/cpythonbuild:ubuntu-impish-aarch64

...
aarch64-linux-gnu-gcc -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I/cpython/Include/internal -I/usr/include/x86_64-linux-gnu -I. -IObjects -IPython -I/usr/include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/cpython/Include -I/cpython/builddep/ubuntu-impish-x86_64 -c /cpython/Modules/_ctypes/_ctypes.c -o build/temp.linux-aarch64-3.10/cpython/Modules/_ctypes/_ctypes.o -DPy_BUILD_CORE_MODULE -DHAVE_FFI_PREP_CIF_VAR=1 -DHAVE_FFI_PREP_CLOSURE_LOC=1 -DHAVE_FFI_CLOSURE_ALLOC=1
In file included from /usr/aarch64-linux-gnu/include/features.h:508,
                 from /usr/include/x86_64-linux-gnu/bits/libc-header-start.h:33,
                 from /usr/aarch64-linux-gnu/include/limits.h:26,
                 from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:203,
                 from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/syslimits.h:7,
                 from /usr/lib/gcc-cross/aarch64-linux-gnu/11/include/limits.h:34,
                 from /cpython/Include/Python.h:11,
                 from /cpython/Modules/zlibmodule.c:8:
/usr/include/x86_64-linux-gnu/gnu/stubs.h:7:11: fatal error: gnu/stubs-32.h: No such file or directory
    7 | # include <gnu/stubs-32.h>
msg406935 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-24 16:13
bpo-45886 addresses the cross build issue with freeze_module command.

The wrong header files come from the fact that setup.py uses CC variable from sysconfig instead of environment. The sysconfig variable contains the C compiler of the build interpreter instead of the host interpreter. The correct value is in os.environ.
msg406942 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-24 17:53
New changeset b30bf4520ae9d6e7eca09d812dd8a86c020b9202 by Christian Heimes in branch 'main':
bpo-45881: Use CC from env first for cross building (GH-29752)
https://github.com/python/cpython/commit/b30bf4520ae9d6e7eca09d812dd8a86c020b9202
msg407016 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 19:53
New changeset cd6d2577fadc4cc0275017f27f46b0a628216353 by Christian Heimes in branch '3.9':
[3.9] bpo-45881: Use CC from env first for cross building (GH-29752) (GH-29754)
https://github.com/python/cpython/commit/cd6d2577fadc4cc0275017f27f46b0a628216353
msg407165 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-27 20:14
New changeset 545aebd2ecef9f6c3b2ca1973e3e0515d8355ce3 by Christian Heimes in branch '3.10':
[3.10] bpo-45881: Use CC from env first for cross building (GH-29752). (GH-29753)
https://github.com/python/cpython/commit/545aebd2ecef9f6c3b2ca1973e3e0515d8355ce3
msg407292 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-29 16:23
New changeset 992565f7f72fd8250b788795f76eedcff5636a64 by Christian Heimes in branch 'main':
bpo-45881: configure --with-freeze-module --with-build-python (GH-29835)
https://github.com/python/cpython/commit/992565f7f72fd8250b788795f76eedcff5636a64
History
Date User Action Args
2021-11-29 16:23:38christian.heimessetmessages: + msg407292
2021-11-29 11:45:56christian.heimessetpull_requests: + pull_request28066
2021-11-27 20:14:12christian.heimessetmessages: + msg407165
2021-11-27 12:06:24christian.heimessetdependencies: + sysconfig --generate-posix-vars creates wrong file when cross compiling
2021-11-26 09:41:56christian.heimeslinkissue40280 dependencies
2021-11-25 19:53:23christian.heimessetmessages: + msg407016
2021-11-24 18:01:05christian.heimessetpull_requests: + pull_request27991
2021-11-24 17:57:55christian.heimessetpull_requests: + pull_request27990
2021-11-24 17:53:41christian.heimessetmessages: + msg406942
2021-11-24 16:18:41christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request27989
2021-11-24 16:13:08christian.heimessetdependencies: + Fix Program/_freeze_module for cross compiling Python
messages: + msg406935
2021-11-23 17:00:57christian.heimessetmessages: + msg406860
2021-11-23 16:37:45christian.heimescreate