Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

De-couple the Windows builds from freezing modules. #89351

Closed
ericsnowcurrently opened this issue Sep 13, 2021 · 10 comments
Closed

De-couple the Windows builds from freezing modules. #89351

ericsnowcurrently opened this issue Sep 13, 2021 · 10 comments
Labels
3.11 only security fixes build The build process and cross-build OS-windows type-bug An unexpected behavior, bug, or error

Comments

@ericsnowcurrently
Copy link
Member

BPO 45188
Nosy @gvanrossum, @pfmoore, @tjguk, @ericsnowcurrently, @zware, @zooba
PRs
  • bpo-45188: Windows now regenerates frozen modules at the start of build instead of late #28322
  • bpo-45020: Freeze some of the modules imported during startup. #28335
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-09-16.01:15:56.780>
created_at = <Date 2021-09-13.20:44:06.042>
labels = ['type-bug', 'OS-windows', 'build', '3.11']
title = 'De-couple the Windows builds from freezing modules.'
updated_at = <Date 2021-09-16.01:15:56.779>
user = 'https://github.com/ericsnowcurrently'

    bugs.python.org fields:

    activity = <Date 2021-09-16.01:15:56.779>
actor = 'eric.snow'
assignee = 'none'
closed = True
closed_date = <Date 2021-09-16.01:15:56.780>
closer = 'eric.snow'
components = ['Build', 'Windows']
creation = <Date 2021-09-13.20:44:06.042>
creator = 'eric.snow'
dependencies = []
files = []
hgrepos = []
issue_num = 45188
keywords = ['patch']
message_count = 10.0
messages = ['401731', '401735', '401849', '401859', '401868', '401877', '401882', '401907', '401908', '401909']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'paul.moore', 'tim.golden', 'eric.snow', 'zach.ware', 'steve.dower']
pr_nums = ['28322', '28335']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue45188'
versions = ['Python 3.11']
    @ericsnowcurrently
    Copy link
    Member Author

    Currently for Windows builds, generating the frozen modules depends on first building python.exe. One consequence of this is that we must keep all frozen module .h files in the repo (which we'd like to avoid for various reasons).

    We should be able to freeze modules before building python.exe, like we already do via our Makefile. From what I understand, this will require that a subset of the runtime be separately buildable so we can use it in _freeze_module.c and use that before actually building python.exe.

    @Steve, please correct any details I got wrong here. :)

    @ericsnowcurrently ericsnowcurrently added 3.11 only security fixes build The build process and cross-build type-bug An unexpected behavior, bug, or error labels Sep 13, 2021
    @zooba
    Copy link
    Member

    zooba commented Sep 13, 2021

    Only thing I'd add is that you should just be able to list the required .c files in _freeze_module.vcxproj (formerly known as freeze_importlib.vcxproj) rather than depending on pythoncore.vcxproj.

    That will generate twice as many .obj files for those modules (which is fine, just takes a little more time at build), and will force everything to be regenerated if you modify them, but that's an expected part of having part of the interpreter depend upon itself.

    @ericsnowcurrently
    Copy link
    Member Author

    New changeset cbeb819 by Eric Snow in branch 'main':
    bpo-45020: Freeze some of the modules imported during startup. (gh-28335)
    cbeb819

    @zooba
    Copy link
    Member

    zooba commented Sep 15, 2021

    New changeset 09b4ad1 by Steve Dower in branch 'main':
    bpo-45188: Windows now regenerates frozen modules at the start of build instead of late (GH-28322)
    09b4ad1

    @gvanrossum
    Copy link
    Member

    Is this now done? I.e. can we now drop the frozen .h files from the repo?

    @zooba
    Copy link
    Member

    zooba commented Sep 15, 2021

    Should be able to, yeah. Though I didn't test that.

    @gvanrossum
    Copy link
    Member

    I tried this:

    • remove the generated .h files
    • touched dictobject.c
    • touched dictobject.h

    After each step I tried to rebuild. Each case the compilation of frozen.c failed and then the build stopped, so apparently the .h files weren't generated early enough.

    Here's the error message:

    C:\Users\gvanrossum\cpython\Python\frozen.c(44,10): fatal error C1083: Cannot open include file: 'frozen_modules/abc.
    h': No such file or directory [C:\Users\gvanrossum\cpython\PCbuild\pythoncore.vcxproj]

    @ericsnowcurrently
    Copy link
    Member Author

    FYI, I have a PR up for dropping the .h files: #28375.

    @ericsnowcurrently
    Copy link
    Member Author

    Can we close this?

    @gvanrossum
    Copy link
    Member

    We can once #72561 lands.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.11 only security fixes build The build process and cross-build OS-windows type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    3 participants
    @ericsnowcurrently @zooba @gvanrossum