classification
Title: Fix Program/_freeze_module for cross compiling Python
Type: enhancement Stage: patch review
Components: Build Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, eric.snow, gvanrossum, iritkatriel
Priority: normal Keywords: patch

Created on 2021-11-23 22:35 by christian.heimes, last changed 2021-11-29 12:45 by christian.heimes.

Pull Requests
URL Status Linked Edit
PR 29735 merged christian.heimes, 2021-11-23 22:38
PR 29793 closed christian.heimes, 2021-11-26 12:52
Messages (8)
msg406893 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-23 22:35
The build process creates the binary Program/_freeze_module, which is used to create frozen modules. The program takes a Python file, compiles it, and stores its marshalled output in a header file.

The approach does not work when cross compiling Python. In cross building case Program/_freeze_module cannot be executed on the build host. For example a cross build with build host "x86_64" and target host "aarch64" would create a aarch64 Program/_freeze_module. The current x86_64 host cannot executed binary (unless you use qemu and binfmt, which I'm deliberately ignoring here).

To unblock cross building and until we find a better solution, I propose that we allow developers to override the freeze module command on the command line. This allows developers to use a freeze_module program from a non-cross build:

   ../../configure -C --host=aarch64-linux-gnu-gcc --build=x86_64-pc-linux-gnu
   make FREEZE_MODULE=../x86_64/Program/_freeze_module
msg406898 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-11-23 23:00
To the question "why does _freeze_module need to be a (C) binary" (IOW, why can't it be replaced by a Python script that is run with PYTHON_FOR_REGEN, which always runs on the build host), the reason is that it uses the bytecode compiler to generate correct output. The bytecode compiler is written in C and depends on many other parts of the runtime. Bytecode is not compatible between versions (and in 3.11 it is an especially moving target) so we absolutely must use the bytecode compiler from the current source tree.

Fortunately bytecode *is* portable across OS and CPU versions, so there is no problem with taking a _freeze_module binary compiled for the build host and running it there (on the build host). It *is* complicated to build a binary for the build host in a tree configured for cross compilation though -- you'd have to do an out-of-tree build configured for the build host platform. Rather than solving *that* problem, Christian proposes to let the user solve that, and allowing the user to pass in the path to the host platform _freeze_module binary.
msg406904 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-24 07:07
New changeset dd8ce9ea8d7e45224a348ea7962b5f50cbf86dc5 by Christian Heimes in branch 'main':
bpo-45886: Allow overriding freeze command for cross compiling (GH-29735)
https://github.com/python/cpython/commit/dd8ce9ea8d7e45224a348ea7962b5f50cbf86dc5
msg406988 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-25 11:25
The FREEZE_MODULE trick fails with out-of-tree builds when SRCDIR contains the frozen_modules header files from a previous build. When doing OOT building, make includes the SRCDIR in the search path for dependencies (VPATH). It considers $(srcdir)/Python/frozen_modules/importlib._bootstrap_external.h as recent enough and therefore does not generate $(builddir)/Python/frozen_modules/importlib._bootstrap_external.h. The freeze_module program does not know about VPATH and looks up the file in the $(builddir) tree:

$ make FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module
...
python3.9 ../../Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap_external.h -m importlib._bootstrap_external -o Python/deepfreeze/importlib._bootstrap_external.c
Traceback (most recent call last):
  File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 463, in <module>
    main()
  File "/cpython/builddep/wasi/../../Tools/scripts/deepfreeze.py", line 451, in main
    with open(args.file, encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'Python/frozen_modules/importlib._bootstrap_external.h'
make: *** [Makefile:1055: Python/deepfreeze/importlib._bootstrap_external.c] Error 1

$ make FREEZE_MODULE=/cpython/builddep/ubuntu-impish-x86_64/Programs/_freeze_module Python/frozen_modules/importlib._bootstrap_external.h  
make: '../../Python/frozen_modules/importlib._bootstrap_external.h' is up to date.


I see two possible solutions for the problem:

* extend ``make clean`` to also remove $(FROZEN_FILES_OUT). The make clean command is suppose to remove all files that interferes with OOT builds (see check-clean-src in Makefile.pre.in).
* prefix $(FROZEN_FILES_OUT) and their targets with $(abs_builddir) to avoid VPATH lookups. abs_builddir is the absolute path to the build directory.
msg407006 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-11-25 15:58
Imthink I prefer the ‘make clean’ extension.--
--Guido (mobile)
msg407062 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-26 16:05
New changeset 765b2a3ad2e8abf9a06d5e9b3802b575ec115d76 by Christian Heimes in branch 'main':
bpo-45886: Fix OOT build when srcdir has frozen module headers (GH-29793)
https://github.com/python/cpython/commit/765b2a3ad2e8abf9a06d5e9b3802b575ec115d76
msg407267 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-11-29 11:59
Is this related to this issue or something else?

https://buildbot.python.org/all/#/builders/310/builds/344

gcc -pthread -c -Wsign-compare -g -Og -Wall    -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE -o Programs/_freeze_module.o Programs/_freeze_module.c
gcc -pthread     -o Programs/_freeze_module Programs/_freeze_module.o Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o    Modules/config.o Modules/getpath.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o -lpthread -ldl  -lutil                        -lm 
Programs/_freeze_module importlib._bootstrap ./Lib/importlib/_bootstrap.py Python/frozen_modules/importlib._bootstrap.h
python3 ./Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap.h -m importlib._bootstrap -o Python/deepfreeze/importlib._bootstrap.c
  File "./Tools/scripts/deepfreeze.py", line 29
    res.append(f"\\x{i:02x}")
                           ^
SyntaxError: invalid syntax
Makefile:1069: recipe for target 'Python/deepfreeze/importlib._bootstrap.c' failed
make: *** [Python/deepfreeze/importlib._bootstrap.c] Error 1
program finished with exit code 2
elapsedTime=45.181814
msg407272 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-11-29 12:45
The issue is related to bpo-45873. The Debian buildbot has an ancient Python version that does not support f-strings.
History
Date User Action Args
2021-11-29 12:45:22christian.heimessetmessages: + msg407272
2021-11-29 11:59:24iritkatrielsetnosy: + iritkatriel
messages: + msg407267
2021-11-26 16:05:25christian.heimessetmessages: + msg407062
2021-11-26 12:52:15christian.heimessetpull_requests: + pull_request28028
2021-11-26 09:41:56christian.heimeslinkissue40280 dependencies
2021-11-25 15:58:56gvanrossumsetmessages: + msg407006
2021-11-25 11:25:01christian.heimessetmessages: + msg406988
2021-11-24 16:13:08christian.heimeslinkissue45881 dependencies
2021-11-24 07:07:23christian.heimessetmessages: + msg406904
2021-11-23 23:00:08gvanrossumsetnosy: + eric.snow, gvanrossum
messages: + msg406898
2021-11-23 22:38:23christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request27973
2021-11-23 22:35:22christian.heimescreate