This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Share global string identifiers in deepfreeze
Type: resource usage Stage: resolved
Components: Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Dennis Sweeney, eric.snow, gvanrossum, kumaraditya
Priority: normal Keywords: patch

Created on 2022-02-10 14:18 by kumaraditya, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 31261 merged kumaraditya, 2022-02-10 14:27
PR 31596 open kumaraditya, 2022-02-26 07:22
PR 31637 merged eric.snow, 2022-03-01 17:31
PR 32061 merged eric.snow, 2022-03-22 22:53
PR 32152 open kumaraditya, 2022-03-28 09:14
Messages (7)
msg413003 - (view) Author: Kumar Aditya (kumaraditya) * (Python triager) Date: 2022-02-10 14:18
Since bpo-46541, the global strings are statically allocated so they can now be referenced by deep-frozen modules just like any other singleton. Sharing identifiers with deepfreeze will reduce the duplicated strings hence it would save space.

See https://github.com/faster-cpython/ideas/issues/218
See https://github.com/faster-cpython/ideas/issues/230
msg413005 - (view) Author: Kumar Aditya (kumaraditya) * (Python triager) Date: 2022-02-10 14:27
I have refactored generate_global_objects.py, and now instead of hard-coding every identifier manually, it now scans *.c files extracts the identifiers used in it and then generate the header file. This has multiple advantages:

- No need to manually add identifiers, as soon as it is used in a c file it is added to the global identifiers struct.
- It simplifies the codegen a lot.
- Remove the need of special casing certain file for checking now it is just a set of identifiers and auto removes unused global strings.
msg414028 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-02-25 18:05
New changeset eb002dbe0da9622245a355db5f0cd5aa2fc70b40 by Kumar Aditya in branch 'main':
bpo-46712: Share global string identifiers in deepfreeze (GH-31261)
https://github.com/python/cpython/commit/eb002dbe0da9622245a355db5f0cd5aa2fc70b40
msg414295 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-03-01 21:30
New changeset 21099fc064c61d59c936a2f6a0db3e07cd5c8de5 by Eric Snow in branch 'main':
bpo-46712: Let generate_global_objects.py Run on Earlier Python Versions (gh-31637)
https://github.com/python/cpython/commit/21099fc064c61d59c936a2f6a0db3e07cd5c8de5
msg415665 - (view) Author: Dennis Sweeney (Dennis Sweeney) * (Python committer) Date: 2022-03-21 04:49
In bpo-47067, there was concern about the addition of the makefile target from PR 31637:

    regen-global-objects: regen-deepfreeze

After a new `&_Py_ID(__orig_class__)` is added to Objects/genericaliasobject.c, running `make regen-global-objects` starts

    gcc -pthread -c [snipped] -DPy_BUILD_CORE -o Objects/genericaliasobject.o Objects/genericaliasobject.c

which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince `make` to regenerate the global objects without this sort of circular dependency? Am I missing a step?
msg415811 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-03-22 20:50
> After a new `&_Py_ID(__orig_class__)` is added to Objects/genericaliasobject.c, running `make regen-global-objects` starts
>
>     gcc -pthread -c [snipped] -DPy_BUILD_CORE -o Objects/genericaliasobject.o Objects/genericaliasobject.c
>
> which fails with a compilation error because that identifier is not yet defined. Is there a good way to convince `make` to regenerate the global objects without this sort of circular dependency? Am I missing a step?

I'm looking into this.  A temporary workaround is to run Tools/scripts/generate-global-objects.py directly.
msg415882 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-03-23 15:56
New changeset febf54bcf3fdc45ad84b4073e24bbaaee0ac8b2a by Eric Snow in branch 'main':
bpo-46712: Do not Regen Deep-Frozen Modules before Generating Global Objects (gh-32061)
https://github.com/python/cpython/commit/febf54bcf3fdc45ad84b4073e24bbaaee0ac8b2a
History
Date User Action Args
2022-04-11 14:59:56adminsetgithub: 90868
2022-03-28 09:14:38kumaradityasetpull_requests: + pull_request30230
2022-03-23 15:56:00eric.snowsetmessages: + msg415882
2022-03-22 22:53:09eric.snowsetpull_requests: + pull_request30151
2022-03-22 20:50:57eric.snowsetmessages: + msg415811
2022-03-21 04:49:39Dennis Sweeneysetnosy: + Dennis Sweeney
messages: + msg415665
2022-03-01 21:30:22eric.snowsetmessages: + msg414295
2022-03-01 17:31:17eric.snowsetpull_requests: + pull_request29759
2022-02-26 07:22:28kumaradityasetpull_requests: + pull_request29718
2022-02-25 18:08:29gvanrossumsetstatus: open -> closed
type: resource usage
resolution: fixed
stage: patch review -> resolved
2022-02-25 18:05:39gvanrossumsetmessages: + msg414028
2022-02-10 14:27:52kumaradityasetkeywords: + patch
stage: patch review
pull_requests: + pull_request29427
2022-02-10 14:27:25kumaradityasetnosy: + eric.snow
messages: + msg413005
2022-02-10 14:18:03kumaradityacreate