New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix Program/_freeze_module for cross compiling Python #90044
Comments
The build process creates the binary Program/_freeze_module, which is used to create frozen modules. The program takes a Python file, compiles it, and stores its marshalled output in a header file. The approach does not work when cross compiling Python. In cross building case Program/_freeze_module cannot be executed on the build host. For example a cross build with build host "x86_64" and target host "aarch64" would create a aarch64 Program/_freeze_module. The current x86_64 host cannot executed binary (unless you use qemu and binfmt, which I'm deliberately ignoring here). To unblock cross building and until we find a better solution, I propose that we allow developers to override the freeze module command on the command line. This allows developers to use a freeze_module program from a non-cross build: ../../configure -C --host=aarch64-linux-gnu-gcc --build=x86_64-pc-linux-gnu |
To the question "why does _freeze_module need to be a (C) binary" (IOW, why can't it be replaced by a Python script that is run with PYTHON_FOR_REGEN, which always runs on the build host), the reason is that it uses the bytecode compiler to generate correct output. The bytecode compiler is written in C and depends on many other parts of the runtime. Bytecode is not compatible between versions (and in 3.11 it is an especially moving target) so we absolutely must use the bytecode compiler from the current source tree. Fortunately bytecode *is* portable across OS and CPU versions, so there is no problem with taking a _freeze_module binary compiled for the build host and running it there (on the build host). It *is* complicated to build a binary for the build host in a tree configured for cross compilation though -- you'd have to do an out-of-tree build configured for the build host platform. Rather than solving *that* problem, Christian proposes to let the user solve that, and allowing the user to pass in the path to the host platform _freeze_module binary. |
The FREEZE_MODULE trick fails with out-of-tree builds when SRCDIR contains the frozen_modules header files from a previous build. When doing OOT building, make includes the SRCDIR in the search path for dependencies (VPATH). It considers $ make FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module
...
python3.9 ../../Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap_external.h -m importlib._bootstrap_external -o Python/deepfreeze/importlib._bootstrap_external.c
Traceback (most recent call last):
File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 463, in <module>
main()
File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 451, in main
with open(args.file, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'Python/frozen_modules/importlib._bootstrap_external.h'
make: *** [Makefile:1055: Python/deepfreeze/importlib._bootstrap_external.c] Error 1 $ make FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module Python/frozen_modules/importlib._bootstrap_external.h
make: '../../Python/frozen_modules/importlib._bootstrap_external.h' is up to date. I see two possible solutions for the problem:
|
Imthink I prefer the ‘make clean’ extension.-- |
The issue is related to bpo-45873. The Debian buildbot has an ancient Python version that does not support f-strings. |
Can this be closed now ? |
I guess so. (@tiran If there's still something to do here feel free to reopen.) |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: