This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [meta] Freeze commonly used stdlib modules.
Type: behavior Stage:
Components: Build Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: FFY00, christian.heimes, eric.snow, gvanrossum, kumaraditya, methane
Priority: normal Keywords:

Created on 2021-10-28 19:39 by eric.snow, last changed 2022-04-11 14:59 by admin.

Messages (7)
msg405239 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-28 19:39
We're already freezing modules needed for bootstrapping the runtime, as well as the modules imported during startup. [1][2]  We're also planning on freezing argparse. [3]  There may be other modules (or files) worth freezing (assuming we don't just freeze the entire stdlib, which would have some potential downsides).

This issue is meant to cover the broader idea.  The work to actually freeze modules should be handled in separate issues.


[1] https://bugs.python.org/issue45020
[2] https://bugs.python.org/issue45654
[3] https://bugs.python.org/issue45660
msg405240 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-28 19:44
One to consider that isn't necessarily used that commonly is sysconfig.  However, that could also involve freezing the "sysconfigdata" file (without needing to expose it as a module).
msg409947 - (view) Author: Kumar Aditya (kumaraditya) * (Python triager) Date: 2022-01-07 04:58
functools and contextlib are very common so they should be deep-frozen. warnings and re would be nice to do.
msg409948 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2022-01-07 06:47
I don't against deep freezing functools and contextlib.
But I think we should optimize and utilize zipimport or something similar, because we can not deep-freeze all stdlib or 3rd party libraries.

See also: https://github.com/faster-cpython/ideas/discussions/158#discussioncomment-1857198
msg409958 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2022-01-07 11:06
Deep freezing is not a miracle cure for startup performance issues. Every additional frozen module increases the size of the main binary / shared library. This can have negative affects for performance and usability, too. For WASM builds we want to make the wasm binary as small as possible.
msg409959 - (view) Author: Kumar Aditya (kumaraditya) * (Python triager) Date: 2022-01-07 11:15
Can you elaborate a bit about "This can have negative affects for performance" as I think freezing modules has only one downside of increasing binary size ?

> For WASM builds we want to make the wasm binary as small as possible.

If WASM would become an officially supported platform then there can be a patch to disable freezing on such platforms otherwise I don't see how because of WASM we cannot deepfreeze for other platforms which benefit from it.
msg410024 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-01-07 20:05
Deep-freezing is definitely not a miracle cure. We should continue to trim or delay unneeded imports (like Christian just did for setuptools' _distutils_hack).

What *would* be a miracle cure would be if we could deep-freeze the module contents after it has *executed*. But that is much harder in general, since two executions (even on the same platform) could result in different module contents (e.g. conditionally defining something based on environment contents).

Instagram's Cinder has something that works for this, strict modules (https://github.com/facebookincubator/cinder#strict-modules). It is a complex system! But if we could do this it could really speed up a lot of imports tremendously. (Though still at the cost of binary size increase.)
History
Date User Action Args
2022-04-11 14:59:51adminsetgithub: 89824
2022-01-07 20:05:59gvanrossumsetmessages: + msg410024
2022-01-07 11:15:46kumaradityasetmessages: + msg409959
2022-01-07 11:06:51christian.heimessetnosy: + christian.heimes
messages: + msg409958
2022-01-07 06:47:07methanesetnosy: + methane
messages: + msg409948
2022-01-07 04:58:28kumaradityasetnosy: + kumaraditya, gvanrossum
messages: + msg409947
2021-10-28 19:44:19eric.snowsetmessages: + msg405240
2021-10-28 19:39:40eric.snowcreate