classification
Title: De-couple the Windows builds from freezing modules.
Type: behavior Stage: resolved
Components: Build, Windows Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, gvanrossum, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2021-09-13 20:44 by eric.snow, last changed 2021-09-16 01:15 by eric.snow. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28322 merged steve.dower, 2021-09-13 23:51
PR 28335 merged eric.snow, 2021-09-14 16:09
Messages (10)
msg401731 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-13 20:44
Currently for Windows builds, generating the frozen modules depends on first building python.exe.  One consequence of this is that we must keep all frozen module .h files in the repo (which we'd like to avoid for various reasons).

We should be able to freeze modules before building python.exe, like we already do via our Makefile.  From what I understand, this will require that a subset of the runtime be separately buildable so we can use it in _freeze_module.c and use that before actually building python.exe.

@Steve, please correct any details I got wrong here. :)
msg401735 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-13 20:59
Only thing I'd add is that you should just be able to list the required .c files in _freeze_module.vcxproj (formerly known as freeze_importlib.vcxproj) rather than depending on pythoncore.vcxproj.

That will generate twice as many .obj files for those modules (which is fine, just takes a little more time at build), and will force everything to be regenerated if you modify them, but that's an expected part of having part of the interpreter depend upon itself.
msg401849 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-15 16:19
New changeset cbeb81971057d6c382f45ecce92df2b204d4106a by Eric Snow in branch 'main':
bpo-45020: Freeze some of the modules imported during startup. (gh-28335)
https://github.com/python/cpython/commit/cbeb81971057d6c382f45ecce92df2b204d4106a
msg401859 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-15 17:11
New changeset 09b4ad11f323f8702cde795e345b75e0fbb1a9a5 by Steve Dower in branch 'main':
bpo-45188: Windows now regenerates frozen modules at the start of build instead of late (GH-28322)
https://github.com/python/cpython/commit/09b4ad11f323f8702cde795e345b75e0fbb1a9a5
msg401868 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-09-15 17:56
Is this now done? I.e. can we now drop the frozen .h files from the repo?
msg401877 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-09-15 18:32
Should be able to, yeah. Though I didn't test that.
msg401882 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-09-15 18:49
I tried this:

- remove the generated .h files
- touched dictobject.c
- touched dictobject.h

After each step I tried to rebuild. Each case the compilation of frozen.c failed and then the build stopped, so apparently the .h files weren't generated early enough.

Here's the error message:

C:\Users\gvanrossum\cpython\Python\frozen.c(44,10): fatal error C1083: Cannot open include file: 'frozen_modules/abc.
h': No such file or directory [C:\Users\gvanrossum\cpython\PCbuild\pythoncore.vcxproj]
msg401907 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-15 21:08
FYI, I have a PR up for dropping the .h files: https://github.com/python/cpython/pull/28375.
msg401908 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-09-15 21:08
Can we close this?
msg401909 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-09-15 21:12
We can once GH-28375 lands.
History
Date User Action Args
2021-09-16 01:15:56eric.snowsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-09-15 21:12:26gvanrossumsetmessages: + msg401909
2021-09-15 21:08:37eric.snowsetmessages: + msg401908
2021-09-15 21:08:16eric.snowsetmessages: + msg401907
2021-09-15 18:49:26gvanrossumsetmessages: + msg401882
2021-09-15 18:32:27steve.dowersetmessages: + msg401877
2021-09-15 17:56:13gvanrossumsetnosy: + gvanrossum
messages: + msg401868
2021-09-15 17:11:19steve.dowersetmessages: + msg401859
2021-09-15 16:19:37eric.snowsetmessages: + msg401849
2021-09-14 16:09:35eric.snowsetpull_requests: + pull_request26747
2021-09-13 23:51:43steve.dowersetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request26731
2021-09-13 20:59:55steve.dowersetnosy: + paul.moore, tim.golden, zach.ware
messages: + msg401735
components: + Windows
2021-09-13 20:48:21eric.snowlinkissue45020 dependencies
2021-09-13 20:44:06eric.snowcreate