classification
Title: Reintroduce bootstrap_python for freezing
Type: behavior Stage: patch review
Components: Build Versions: Python 3.11
process
Status: open Resolution:
Dependencies: 45696 45873 Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, eric.snow, gvanrossum, ned.deily, pablogsal, ronaldoussoren
Priority: Keywords: patch

Created on 2021-12-01 16:26 by christian.heimes, last changed 2021-12-07 02:25 by ned.deily.

Pull Requests
URL Status Linked Edit
PR 29859 merged christian.heimes, 2021-12-01 16:29
PR 29936 closed christian.heimes, 2021-12-06 10:02
PR 29939 closed christian.heimes, 2021-12-06 13:17
PR 29939 closed christian.heimes, 2021-12-06 13:17
Messages (11)
msg407467 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-01 16:26
bpo-45696 introduced a _bootstrap_python interpreter, which was used to create frozen and deepfrozen module files. bpo-45873 dropped the _bootstrap_python interpreter again. Instead Python used an existing Python installation to bootstrap the build.

After some internal discussion we agreed that the approach is flawed and puts too much of a burden on users. Users have to install a fairly recent Python interpreter in order to build 3.11. This makes it harder to build Python on platforms with no or a too old Python version. It also complicated things for container builds and pyenv.

Let's re-introduce a _bootstrap_python interpreter. Cross compiling will still require a build Python interpreter.
msg407592 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-03 15:01
New changeset 84ca1232b0f1e4be368e89550a9ceb46f64a0eff by Christian Heimes in branch 'main':
bpo-45950: Introduce Bootstrap Python again (#29859)
https://github.com/python/cpython/commit/84ca1232b0f1e4be368e89550a9ceb46f64a0eff
msg407776 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-12-06 09:10
Framework builds on macOS now fail with a bus error when trying to run the newly built _bootstrap_python. A simplified example:

$ ./configure --enable-framework=$PWD/Library/Frameworks
checking for git... found
checking build system type... x86_64-apple-darwin20.6.0
checking host system type... x86_64-apple-darwin20.6.0
checking for Python interpreter freezing... ./_bootstrap_python
checking for python3.11... no
checking for python3.10... no
checking for python3.9... no
checking for python3.8... no
checking for python3.7... no
checking for python3.6... no
checking for python3... python3
checking Python for regen version... Python 3.8.9
[...]
creating Modules/Setup.local
creating Makefile

pkg-config is missing. Some dependencies may not be detected correctly.


If you want a release build with all stable optimizations active (PGO, etc),
please run ./configure --enable-optimizations


$ make
[...]
gcc     -o _bootstrap_python Modules/getbuildinfo.o Parser/token.o  Parser/pegen.o Parser/pegen_errors.o Parser/action_helpers.o Parser/parser.o Parser/string_parser.o Parser/peg_api.o Parser/myreadline.o Parser/tokenizer.o Objects/abstract.o Objects/accu.o Objects/boolobject.o Objects/bytes_methods.o Objects/bytearrayobject.o Objects/bytesobject.o Objects/call.o Objects/capsule.o Objects/cellobject.o Objects/classobject.o Objects/codeobject.o Objects/complexobject.o Objects/descrobject.o Objects/enumobject.o Objects/exceptions.o Objects/genericaliasobject.o Objects/genobject.o Objects/fileobject.o Objects/floatobject.o Objects/frameobject.o Objects/funcobject.o Objects/interpreteridobject.o Objects/iterobject.o Objects/listobject.o Objects/longobject.o Objects/dictobject.o Objects/odictobject.o Objects/memoryobject.o Objects/methodobject.o Objects/moduleobject.o Objects/namespaceobject.o Objects/object.o Objects/obmalloc.o Objects/picklebufobject.o Objects/rangeobject.o Objects/setobject.o Objects/sliceobject.o Objects/structseq.o Objects/tupleobject.o Objects/typeobject.o Objects/unicodeobject.o Objects/unicodectype.o Objects/unionobject.o Objects/weakrefobject.o Python/_warnings.o Python/Python-ast.o Python/Python-tokenize.o Python/asdl.o Python/ast.o Python/ast_opt.o Python/ast_unparse.o Python/bltinmodule.o Python/ceval.o Python/codecs.o Python/compile.o Python/context.o Python/dynamic_annotations.o Python/errors.o Python/frame.o Python/frozenmain.o Python/future.o Python/getargs.o Python/getcompiler.o Python/getcopyright.o Python/getplatform.o Python/getversion.o Python/hamt.o Python/hashtable.o Python/import.o Python/importdl.o Python/initconfig.o Python/marshal.o Python/modsupport.o Python/mysnprintf.o Python/mystrtoul.o Python/pathconfig.o Python/preconfig.o Python/pyarena.o Python/pyctype.o Python/pyfpe.o Python/pyhash.o Python/pylifecycle.o Python/pymath.o Python/pystate.o Python/pythonrun.o Python/pytime.o Python/bootstrap_hash.o Python/specialize.o Python/structmember.o Python/symtable.o Python/sysmodule.o Python/thread.o Python/traceback.o Python/getopt.o Python/pystrcmp.o Python/pystrtod.o Python/pystrhex.o Python/dtoa.o Python/formatter_unicode.o Python/fileutils.o Python/suggestions.o Python/dynload_shlib.o    Modules/config.o Modules/main.o Modules/gcmodule.o Modules/atexitmodule.o  Modules/faulthandler.o  Modules/posixmodule.o  Modules/signalmodule.o  Modules/_tracemalloc.o  Modules/_codecsmodule.o  Modules/_collectionsmodule.o  Modules/errnomodule.o  Modules/_io/_iomodule.o Modules/_io/iobase.o Modules/_io/fileio.o Modules/_io/bytesio.o Modules/_io/bufferedio.o Modules/_io/textio.o Modules/_io/stringio.o  Modules/itertoolsmodule.o  Modules/_sre.o  Modules/_threadmodule.o  Modules/timemodule.o  Modules/_weakref.o  Modules/_abc.o  Modules/_functoolsmodule.o  Modules/_localemodule.o  Modules/_operator.o  Modules/_stat.o  Modules/symtablemodule.o  Modules/pwdmodule.o  Modules/xxsubtype.o \
		Programs/_bootstrap_python.o Modules/getpath.o -ldl  -framework CoreFoundation
./_bootstrap_python ./Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap.h -m importlib._bootstrap -o Python/deepfreeze/importlib._bootstrap.c
make: *** [Python/deepfreeze/importlib._bootstrap.c] Bus error: 10
$

--enable-framework causes a number of changes in interpreter building and initialization. Since, as it stands, _bootstrap_python executable isn't and doesn't need to be built as a framework, there is probably some path confusion in getpath.c or elsewhere.

Prior to these changes (at 3.11.0a2 at least), it was still possible to run a framework build executable in the build directory through some careful use of build variables in Makefile (like how "make test" works). Without delving deeper into it at the moment, I'm not sure what to suggest. It may be simpler to just do a clean separate non-framework build to provide _bootstrap_python. And/or possibly re-allow the use of --with-build-python to specify a user-supplied build python. I wonder if there are certain configurations on other platforms where there might be similar problems with the current approach.

I'm marking this as a "release blocker" since it is preventing building the usual python.org macOS binary installer for 3.11.0a3. Since it is still early in the release cycle for 3.11, the release manager might reasonably decide to not delay the release for a fix and thus skip having a macOS installer for a3.
msg407777 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 09:25
Ned, does this change solve the issue with framework builds?

$ git diff
diff --git a/Makefile.pre.in b/Makefile.pre.in
index 8e6e553554d..2068db30855 100644
--- a/Makefile.pre.in
+++ b/Makefile.pre.in
@@ -106,7 +106,7 @@ PY_CFLAGS_NODIST=$(CONFIGURE_CFLAGS_NODIST) $(CFLAGS_NODIST) -I$(srcdir)/Include
 PY_CPPFLAGS=   $(BASECPPFLAGS) -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS)
 PY_LDFLAGS=    $(CONFIGURE_LDFLAGS) $(LDFLAGS)
 PY_LDFLAGS_NODIST=$(CONFIGURE_LDFLAGS_NODIST) $(LDFLAGS_NODIST)
-PY_LDFLAGS_NOLTO=$(PY_LDFLAGS) $(CONFIGURE_LDFLAGS_NOLTO) $(LDFLAGS_NODIST)
+PY_LDFLAGS_NOLTO=$(PY_LDFLAGS_NODIST) $(CONFIGURE_LDFLAGS_NOLTO) $(LDFLAGS_NODIST)
 NO_AS_NEEDED=  @NO_AS_NEEDED@
 CCSHARED=      @CCSHARED@
 # LINKFORSHARED are the flags passed to the $(CC) command that links
msg407779 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-12-06 09:40
> Ned, does this change solve the issue with framework builds?

No. I would guess at a minimum getpath.c for _bootstrap_python might need to be built without WITH_NEXT_FRAMEWORK defined. But I'm out of time for today and don't expect to have much time available in the immediate future.
msg407782 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 10:23
Ronald, Pablo, could you please test GH-29936 on macOS with --enable-framework?
msg407790 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 11:35
If the patch does not help then I can re-enable --with-build-python for non-cross builds.
msg407801 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 13:27
I added --enable-framework=yes to macOS GHA builder.

GH-29939 has ./configure --enable-framework=yes without proposed bootstrap fix. The build is failing early:

./_bootstrap_python ./Tools/scripts/deepfreeze.py Python/frozen_modules/importlib._bootstrap.h -m importlib._bootstrap -o Python/deepfreeze/importlib._bootstrap.c
make: *** [Python/deepfreeze/importlib._bootstrap.c] Bus error: 10


GH-29936 has ./configure --enable-framework=yes and introduces a Modules/getpath_bootstrap.o file. It is the same build as Modules/getpath.o except it does not use the framework feature. The target compiles getpath.c with an additional -DPY_BOOTSTRAP_PYTHON=1 option. The build fails later:

/bin/sh: line 1: 14772 Bus error: 10           DYLD_FRAMEWORK_PATH=/Users/runner/work/cpython/cpython ./python.exe -E -S -m sysconfig --generate-posix-vars
generate-posix-vars failed
msg407827 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 16:14
New changeset 612e59b53f0c730ce1b881f7c08dc6d49f02c123 by Christian Heimes in branch 'main':
bpo-45950: Fix macOS framework builds of _bootstrap_python (GH-29936)
https://github.com/python/cpython/commit/612e59b53f0c730ce1b881f7c08dc6d49f02c123
msg407830 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-12-06 16:22
Ned, I have implemented a fix for _bootstrap_python for framework builds and re-enabled --with-build-python for all builds. Previously I disabled the option for standard builds, because --with-build-python can cause build issues when a user runs ./configure --with-build-python with an older 3.11 alpha build.
msg407889 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-12-07 02:25
I've now verified that the _bootstrap_python fix for framework builds works - removing "release blocker" status. Thanks, Christian!
History
Date User Action Args
2021-12-07 02:25:38ned.deilysetpriority: release blocker ->

messages: + msg407889
2021-12-06 16:22:11christian.heimessetmessages: + msg407830
2021-12-06 16:14:03christian.heimessetmessages: + msg407827
2021-12-06 13:27:11christian.heimessetmessages: + msg407801
2021-12-06 13:17:24christian.heimessetpull_requests: + pull_request28164
2021-12-06 13:17:21christian.heimessetpull_requests: + pull_request28163
2021-12-06 11:35:20christian.heimessetmessages: + msg407790
2021-12-06 10:23:33christian.heimessetnosy: + ronaldoussoren, pablogsal
messages: + msg407782
2021-12-06 10:02:48christian.heimessetpull_requests: + pull_request28160
2021-12-06 09:40:28ned.deilysetmessages: + msg407779
2021-12-06 09:25:31christian.heimessetmessages: + msg407777
2021-12-06 09:10:47ned.deilysetpriority: normal -> release blocker
nosy: + ned.deily
messages: + msg407776

2021-12-03 15:01:19christian.heimessetmessages: + msg407592
2021-12-01 16:34:12christian.heimessetnosy: + gvanrossum, eric.snow
2021-12-01 16:29:12christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request28106
2021-12-01 16:26:54christian.heimessetassignee: christian.heimes
dependencies: + "Deep-freeze": skip the marshal step by generating C code, Stop using bootstrap_python for deep-freeze in UNIX builds
2021-12-01 16:26:37christian.heimescreate