Currently we freeze the 3 main import-related modules into the python binary (along with one test module).  This allows us to bootstrap the import machinery from Python modules.  It also means we get better performance importing those modules.

If we freeze modules that are likely to be used during execution then we get even better startup times.  I'll be putting up a PR that does so, freezing all the modules that are imported during startup.  This could also be done for any stdlib modules that are commonly imported.

(also see #45019 and https://github.com/faster-cpython/ideas/issues/82)

FYI, with my branch I'm getting a 15% improvement to startup for "./python -c pass".

I'm aware of two potentially problematic consequences to this change:

* making changes to those source files for the modules will not be reflected during execution until after "make" is run
* tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) may lose their entry point

For the former, I'm not sure there's a way around it.  We may consider the inconvenience worth it in order to get the performance benefits.

For the latter, the obvious solution is to introduce a startup hook (e.g. on the CLI) like we've talked about doing for years.  (I wasn't able to find previous discussions on that topic after a quick search.)

Not sure whether you are aware, but the PyRun project I'm maintaining
does that and goes beyond this by freezing almost the complete stdlib
and statically linking most C extensions into a single binary:

https://www.egenix.com/products/python/PyRun/

Startup is indeed better, but not as much as you might think.
You do save stat calls and can share resources across processes.

The big time consumer is turning marshal'ed code objects back
into Python objects, though. If that could be made faster by
e.g. using a more efficient storage format such as one which is
platform dependent, it'd be a much bigger win than the freezing
approach.

> The big time consumer is turning marshal'ed code objects back
> into Python objects, though. If that could be made faster by
> e.g. using a more efficient storage format such as one which is
> platform dependent, it'd be a much bigger win than the freezing
> approach.

That's something Guido has been exploring. :)

See: https://github.com/faster-cpython/ideas/issues/84 (and others)

> For the latter, the obvious solution is to introduce a startup hook

I'm not sure why I said "obvious".  Sorry about that.

I noticed nedbat un-nosied himself. Probably he didn't realize you were
calling him out because it's possible this would affect coverage.py?

> The big time consumer is turning marshal'ed code objects back
> into Python objects, though. If that could be made faster by
> e.g. using a more efficient storage format such as one which is
> platform dependent, it'd be a much bigger win than the freezing
> approach.

I've explored a couple of different approaches here (see the issue Eric linked to and a few adjacent ones) and this is a tricky issue. Marshal seems to be pretty darn efficient as a storage format, because it's very compact compared to the Python objects it creates. My final (?) proposal is creating static data structures embedded in the code that just *are* Python objects. Unfortunately on Windows the C compiler balks at this -- the C++ compiler handles it just fine, but it's not clear that we are able to statically link C++ object files into Python without depending on a lot of other C++ infrastructure. (In GCC and Clang this is apparently a language extension.)

@Guido, @Mark Shannon, do you recall the other issue where folks objected to that other patch, due to local changes to source files not being reflected?

Also, one thought that comes to mind is that we could ignore the frozen modules when in a dev environment (and opt in to using the frozen modules when an environment variable).

I don't recall, but...

You can't modify any builtin modules. Freezing modules effectively makes them builtin from a user's perspective. There are plenty of modules that can't be modified:

>>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tokenize', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype')

I don't see why adding a few more modules to that list would be a problem.

Was the objection to feezing *all* modules, not just the core ones?

We should ask Neil S. for the issue where Larry introduced this. That might
have some discussion.

But if I had to guess, it’s confusing that you can see *Python* source that
you can’t edit (or rather, where editing doesn’t get reflected in the next
Python run, unless you also compile it.

I know that occasionally  a debug session I add a print statement to a
stdlib module.
-- 
--Guido (mobile)

Neil, do you recall the story here?

The plot thickens. By searching my extensive GMail archives for Jeethu Rao I found an email from Sept. 14 to python-dev by Larry Hastings titled "Store startup modules as C structures for 20%+ startup speed improvement?"

It references an issue and a PR:

    https://bugs.python.org/issue34690
    https://github.com/python/cpython/pull/9320

Here's a link to the python-dev thread:

    https://mail.python.org/pipermail/python-dev/2018-September/155188.html

There's a lot of discussion there. I'll try to dig through it.

Adding Larry in case he remembers more color. (Larry: the key question here is whether some version of this (like the one I've been working on, or a simpler one that Eric has prepared) is viable, given that any time someone works on one of the frozen or deep-frozen stdlib modules, they will have to run make (with the default target) to rebuild the Python binary with the deep-frozen files.

(Honestly if I were working on any of those modules, I'd just comment out some lines from Eric's freeze_modules.py script and do one rebuild until I was totally satisfied with my work. Either way it's a suboptimal experience for people contributing to those modules. But we stand to gain a ~20% startup time improvement.)

PS. The top comment links to Eric's work.

Since nobody's said so in so many words (so far in this thread anyway): the prototype from Jeethu Rao in 2018 was a different technology than what Eric is doing.  The "Programs/_freeze_importlib.c" Eric's playing with essentially inlines a .pyc file as C static data.  The Jeethu Rao approach is more advanced: instead of serializing the objects, it stores the objects from the .pyc file as pre-initialized C static objects.  So it saves the un-marshalling step, and therefore should be faster.  To import the module you still need to execute the module body code object though--that seems unavoidable.

The python-dev thread covers nearly everything I remember about this.  The one thing I guess I never mentioned is that building and working with the prototype was frightful; it had both Python code and C code, and it was fragile and hard to get working.  My hunch at the time was that it shouldn't be so fragile; it should be possible to write the converter in Python: read in .pyc file, generate .c file.  It might have to make assumptions about the internal structure of the CPython objects it instantiates as C static data, but since we'd ship the tool with CPython this should be only a minor maintenance issue.

In experimenting with the prototype, I observed that simply calling stat() to ensure the frozen .py file hadn't changed on disk lost us about half the performance win from this approach.  I'm not much of a systems programmer, but I wonder if there are (system-proprietary?) library calls one could make to get the stat info for all files in a single directory all at once that might be faster overall.  (Of course, caching this information at startup might make for a crappy experience for people who edit Lib/*.py files while the interpreter is running.)

One more observation about the prototype: it doesn't know how to deal with any mutable types.  marshal.c can deal with list, dict, and set.  Does this matter?  ISTM the tree of objects under a code object will never have a reference to one of these mutable objects, so it's probably already fine.

Not sure what else I can tell you.  It gave us a measurable improvement in startup time, but it seemed fragile, and it was annoying to work with/on, so after hacking on it for a week (at the 2018 core dev sprint in Redmond WA) I put it aside and moved on to other projects.

There should be a boolean flag that enables/disables cached copies of .py files from Lib/.  You should be able to turn it off with either an environment variable or a command-line option, and when it's off it skips all the internal cached stuff and uses the normal .py / .pyc machinery.

With that in place, it'd be great to pre-cache all the .py files automatically read in at startup.

As for changes to the build process: the most analogous thing we have is probably Argument Clinic.  For what it's worth, Clinic hasn't been very well integrated into the CPython build process.  There's a pseudotarget that runs it for you in the Makefile, but it's only ever run manually, and I'm not sure there's *any* build automation for Windows developers.  AFAIK it hasn't really been a problem.  But then I'm not sure this is a very good analogy--the workflow for making Clinic changes is very different from people hacking on Lib/*.py.

It might be sensible to add a mechanism that checks whether or not the pre-cached modules are current.  Store a hash for each cached module and check that they all match.  This could then be part of the release process, run from a GitHub hook, etc.

> Since nobody's said so in so many words (so far in this thread anyway): the prototype from Jeethu Rao in 2018 was a different technology than what Eric is doing.  The "Programs/_freeze_importlib.c" Eric's playing with essentially inlines a .pyc file as C static data.  The Jeethu Rao approach is more advanced: instead of serializing the objects, it stores the objects from the .pyc file as pre-initialized C static objects.  So it saves the un-marshalling step, and therefore should be faster.  To import the module you still need to execute the module body code object though--that seems unavoidable.

Yes, I know. We're discussing two separate ideas -- Eric's approach, which is doing the same we're doing for importlib for more stdlib modules; and "my" approach, dubbed "deep-freeze", which is similar to Jeethu's approach (details in https://github.com/faster-cpython/ideas/issues/84).

What the two approaches have in common is that they require rebuilding the python binary whenever you edit any of the changed modules. I heard somewhere (I'm sorry, I honestly don't recall who said it first, possibly Eric himself) that Jeethu's approach was rejected because of that.

FWIW in my attempts to time this, it looks like the perf benefits of Eric's approach are close to those of deep-freezing. And deep-freezing causes much more bloat of the source code and of the resulting binary. (At runtime the binary size is made up by matching heap savings, but to some people binary size is important too.)

> The python-dev thread covers nearly everything I remember about this.  The one thing I guess I never mentioned is that building and working with the prototype was frightful; it had both Python code and C code, and it was fragile and hard to get working.  My hunch at the time was that it shouldn't be so fragile; it should be possible to write the converter in Python: read in .pyc file, generate .c file.  It might have to make assumptions about the internal structure of the CPython objects it instantiates as C static data, but since we'd ship the tool with CPython this should be only a minor maintenance issue.

Deep-freezing doesn't seem frightful to work with, to me at least. :-) Maybe the foundational work by Eric (e.g. generating sections of Makefile.pre.in) has helped.

I don't understand entirely why Jeethu's prototype had part written in C. I never ran it so I don't know what the generated code looked like, but I have a feeling that for objects that don't reference other objects, it would generate a byte array containing the exact contents of the object structure (which it would get from constructing the object in memory and copying the bytes) which was then put together with the object header (containing the refcount and type) and cast to (PyObject *).

In contrast, for deep-freeze I just reverse engineered what the structures look like and wrote a Python script to generate C code for an initialized instance of those structures. You can look at some examples here: https://github.com/gvanrossum/cpython/blob/codegen/Python/codegen__collections_abc.c . It's verbose but the C compiler handles it just fine (C compilers have evolved to handle *very* large generated programs).

> In experimenting with the prototype, I observed that simply calling stat() to ensure the frozen .py file hadn't changed on disk lost us about half the performance win from this approach.  I'm not much of a systems programmer, but I wonder if there are (system-proprietary?) library calls one could make to get the stat info for all files in a single directory all at once that might be faster overall.  (Of course, caching this information at startup might make for a crappy experience for people who edit Lib/*.py files while the interpreter is running.)

I think the only solution here was hinted at in the python-dev thread from 2018: have a command-line flag to turn it on or off (e.g. -X deepfreeze=1/0) and have a policy for what the default for that flag should be (e.g. on by default in production builds, off by default in developer builds -- anything that doesn't use --enable-optimizations).

> One more observation about the prototype: it doesn't know how to deal with any mutable types.  marshal.c can deal with list, dict, and set.  Does this matter?  ISTM the tree of objects under a code object will never have a reference to one of these mutable objects, so it's probably already fine.

Correct, marshal supports things that you will never see in a code object. Perhaps the reason is that when marshal was invented, it wasn't so clear that code objects should be immutable -- that realization came later, when Greg Stein proposed making them ROM-able. That didn't work out, but the notion that code objects should be strictly mutable (to the python user, at least) was born and is now ingrained.

> Not sure what else I can tell you.  It gave us a measurable improvement in startup time, but it seemed fragile, and it was annoying to work with/on, so after hacking on it for a week (at the 2018 core dev sprint in Redmond WA) I put it aside and moved on to other projects.

I'm not so quick to give up. I do believe I have seen similar startup time improvements. But Eric's version (i.e. this issue) is nearly as good, and the binary bloat is much less -- marshal is way more compact than in-memory objects.

(Second message)

> There should be a boolean flag that enables/disables cached copies of .py files from Lib/.  You should be able to turn it off with either an environment variable or a command-line option, and when it's off it skips all the internal cached stuff and uses the normal .py / .pyc machinery.

Yeah.

> With that in place, it'd be great to pre-cache all the .py files automatically read in at startup.

*All* the .py files? I think the binary bloat cause by deep-freezing the entire stdlib would be excessive. In fact, Eric's approach freezes everything in the encodings package, which turns out to be a lot of files and a lot of code (lots of simple data tables expressed in code), and I found that for basic startup time, it's best not to deep-freeze the encodings module except for __init__.py, aliases.py and utf_8.py.

> As for changes to the build process: the most analogous thing we have is probably Argument Clinic.  For what it's worth, Clinic hasn't been very well integrated into the CPython build process.  There's a pseudotarget that runs it for you in the Makefile, but it's only ever run manually, and I'm not sure there's *any* build automation for Windows developers.  AFAIK it hasn't really been a problem.  But then I'm not sure this is a very good analogy--the workflow for making Clinic changes is very different from people hacking on Lib/*.py.

I think we've got reasonably good automation for both Eric's approach and the deep-freeze approach -- all you need to do is run "make" when you've edited one of the (deep-)frozen modules.

> It might be sensible to add a mechanism that checks whether or not the pre-cached modules are current.  Store a hash for each cached module and check that they all match.  This could then be part of the release process, run from a GitHub hook, etc.

I think the automation that Eric developed is already good enough. (He even generates Windows project files.) See https://github.com/python/cpython/pull/27980 .

> What the two approaches have in common is that they require rebuilding the python binary whenever you edit any of the changed modules. I heard somewhere (I'm sorry, I honestly don't recall who said it first, possibly Eric himself) that Jeethu's approach was rejected because of that.

My dim recollection was that Jeethu's approach wasn't explicitly rejected, more that the community was more "conflicted" than "strongly interested", so I lost interest, and nobody else followed up.


> I don't understand entirely why Jeethu's prototype had part written in C.

My theory: it's easier to serialize C objects from C.  It's maybe even slightly helpful?  But it made building a pain.  And yeah it just doesn't seem necessary.  The code generator will be tied to the C representation no matter how you do it, so you might as well write it in a nice high-level language.


> I never ran it so I don't know what the generated code looked like, [...]

You can see an example of Jeethu's serialized objects here:

https://raw.githubusercontent.com/python/cpython/267c93d61db9292921229fafd895b5ff9740b759/Python/frozenmodules.c

Yours is generally more readable because you're using the new named structure initializers syntax.  Though Jeethu's code is using some symbolic constants (e.g. PyUnicode_1BYTE_KIND) where you're just printing the actual value.


> > With that in place, it'd be great to pre-cache all the .py files automatically read in at startup.
>
> *All* the .py files?  I think the binary bloat cause by deep-freezing the entire stdlib would be excessive.

I did say "all the .py files automatically read in at startup".  In current trunk, there are 32 modules in sys.module at startup (when run non-interactively), and by my count 13 of those are written in Python.

If we go with Eric's approach, that means we'd turn those .pyc files into static data.  My quick experiment suggests that'd be less than 300k.  On my 64-bit Linux system, a default build of current trunk (configure && make -j) yields a 23mb python executable, and a 44mb libpython3.11.a.  If I build without -g, they are 4.3mb and 7mb respectively.  So this speedup would add another 2.5% to the size of a stripped build.

If even that 300k was a concern, the marshal approach would also permit us to compile all the deep-frozen modules into a separate shared library and unload it after we're done.

I don't know what the runtime impact of "deep-freeze" is, but it seems like it'd be pretty minimal.  You're essentially storing these objects in C static data instead of the heap, which should be about the same.  Maybe it'd mean the code objects for the module bodies would stick around longer than they otherwise would?  But that doesn't seem like it'd add that much overhead.

It's interesting to think about applying these techniques to the entire standard library, but as you suggest that would probably be wasteful.  On the other hand: if we made a viable tool that could consume some arbitrary set of .py files and produce a C file, and said C file could then be compiled into a shared library, end users could enjoy this speedup over the subset of the standard library their program used, and perhaps even their own source tree(s).

[Larry]
> The one thing I guess I never mentioned is that building and working with the
> prototype was frightful; it had both Python code and C code, and it was
> fragile and hard to get working.

I took Larry's PR and did a fair amount of cleanup on it to make the build less
painful and fragile.  My branch is fairly easy to re-build.  The major downsides remaining
are that you couldn't update .py files and have them used (static ones
take priority) and the generated C code is quite large.

I didn't make any attempt to work on the serializer, other than to make it work
with an alpha version of Python 3.10.

https://github.com/nascheme/cpython/tree/static_frozen

It was good enough to pass nearly(?) all tests and I did some profiling.  It helped reduce startup time quite a bit.

On 28.08.2021 06:06, Guido van Rossum wrote:
> 
>> With that in place, it'd be great to pre-cache all the .py files automatically read in at startup.
> 
> *All* the .py files? I think the binary bloat cause by deep-freezing the entire stdlib would be excessive. In fact, Eric's approach freezes everything in the encodings package, which turns out to be a lot of files and a lot of code (lots of simple data tables expressed in code), and I found that for basic startup time, it's best not to deep-freeze the encodings module except for __init__.py, aliases.py and utf_8.py.

Eric's approach, as I understand it, is pretty much what PyRun does.
It freezes almost the entire stdlib. The main aim was to save space
and create a Python runtime with very few files for easy installation and
shipment of products written in Python.

For Python 3.8 (I haven't ported it to more recent Python versions yet),
the uncompressed stripped binary is 15MB. UPX compressed, it's only 5MB:

-rwxr-xr-x 1 lemburg lemburg  15M May 19 15:26 pyrun3.8
-rwxr-xr-x 1 lemburg lemburg  32M Aug 26  2020 pyrun3.8-debug
-rwxr-xr-x 1 lemburg lemburg 5.0M May 19 15:26 pyrun3.8-upx

There's no bloat, since you don't need the .py/.pyc files for the stdlib
anymore. In fact, you save quite a bit of disk space compared to a
full Python installation and additionally benefit from the memory
mapping the OS does for sharing access to the marshal'ed byte code
between processes.

That said, some things don't work with such an approach, e.g.
a few packages include additional data files which they expect to
find on disk. Since those are not available anymore, they fail.

For PyRun I have patched some of those packages to include the
data in form of Python modules instead, so that it gets frozen
as well, e.g. the Python grammar files.

Whether this is a good approach for Python in general is a different
question, though. PyRun is created on top of the existing released
Python distribution, so it doesn't optimize for being able to
work with the frozen code. In fact, early versions did not
even have a REPL, since the main point was to run a
single released app.

When I investigated freezing the standard library for PyOxidizer, I ran into a rash of problems. The frozen importer doesn't behave like PathFinder. It doesn't (didn't?) set some common module-level attributes that are documented by the importer "specification" to be set and this failed a handful of tests and lead to runtime issues or breakage in 3rd party packages (such as random packages looking for a __file__ on a common stdlib module).

Also, when I last looked at the CPython source, the frozen importer performed a linear scan of its indexed C array performing strcmp() on each entry until it found what it was looking for. So adding hundreds of modules could result in sufficient overhead and justify using a more efficient lookup algorithm. (PyOxidizer uses Rust's HashMap to index modules by name.)

I fully support more aggressive usage of frozen modules in the standard library to speed up interpreter startup. However, if you want to ship this as enabled by default, from my experience with PyOxidizer, I highly recommend:

* Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems.
* Shoring up the implementation of the frozen importer to make it better conform with the importer specification.
* Be prepared for people to complain about the loss of __file__. e.g. https://github.com/indygreg/PyOxidizer/issues/69 / https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html?highlight=__file__#file-and-cached-module-attributes

Oh, PyOxidizer also ran into more general issues with the frozen importer in that it broke various importlib APIs. e.g. because the frozen importer only supports bytecode, you can't use .__loader__.get_source() to obtain the source of a module. This makes tracebacks more opaque and breaks legitimate API consumers relying on these importlib interfaces.

The fundamental limitations with the frozen importer are why I implemented my own meta path importer (implemented in pure Rust), which is more fully featured, like the PathFinder importer that most people rely on today. That importer is available on PyPI (https://pypi.org/project/oxidized-importer/) and has its own API to facilitate PyOxidizer-like functionality (https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html) if anyone wants to experiment with it.

Gregor, thanks for sharing your experience!

I guess freezing the entire stdlib instead of just a smattering of modules (like we do here) exacerbated the problems in your case.

Builtin modules (such as sys or time) don't have a __file__ attribute either, and nobody has ever complained about this (that I know of). I wonder how far our backwards compatibility guarantee should go -- would this mean we cannot turn any stdlib module written in Python into one written in C (or Rust :-)?

It would be more serious if standard tests fail, but I haven't seen any evidence -- the tests all seem to pass for Eric's PR.

Now, if there are stdlib modules that reference their own __file__ and we want to freeze those, we should switch those to using ResourceReader or importlib.resources of course.

I agree that we should shore up the frozen importer -- probably in a separate PR though. (@Eric: do you think this is worth its own bpo issue?)

I also noticed the linear scan (and it's being called up to 5 times for modules that *aren't* frozen :-). strcmp is *very* fast (isn't it a compiler intrinsic?), but perhaps we should time it and if it seems a problem we could sort the array and do a form of bisection. (A slight problem is that there's an API to mutate the list by changing a pointer, so we may have to detect when the pointer has changed and recompute the size by doing a scan for the sentinel, once.)

Unfortunately I don't think we're yet in a world where we can accept any dependencies on Rust for CPython itself, so we would have to rewrite your example implementations in C if we wanted to use them.

On Fri, Aug 27, 2021 at 6:29 PM Guido van Rossum <report@bugs.python.org> wrote:
> The plot thickens. By searching my extensive GMail archives for Jeethu Rao I found
> an email from Sept. 14 to python-dev by Larry Hastings titled "Store startup modules
> as C structures for 20%+ startup speed improvement?"

Thanks for finding that, Guido!

On Fri, Aug 27, 2021 at 6:37 PM Guido van Rossum <report@bugs.python.org> wrote:
> Either way it's a suboptimal experience for people contributing to those modules. But
> we stand to gain a ~20% startup time improvement.

Agreed, and I think a solution shouldn't be too hard to reach.

On Fri, Aug 27, 2021 at 7:48 PM Larry Hastings <report@bugs.python.org> wrote:
> In experimenting with the prototype, I observed that simply calling stat() to ensure
> the frozen .py file hadn't changed on disk lost us about half the performance win
> from this approach.

Yeah, this is an approach others had suggested and I'd considered.  We
have other solutions available that don't have that penalty.

On Fri, Aug 27, 2021 at 8:08 PM Larry Hastings <report@bugs.python.org> wrote:
> There should be a boolean flag that enables/disables cached copies of .py files from
> Lib/.  You should be able to turn it off with either an environment variable or a
> command-line option, and when it's off it skips all the internal cached stuff and
> uses the normal .py / .pyc machinery.
>
> With that in place, it'd be great to pre-cache all the .py files automatically read in
> at startup.

Yeah, something along these lines should be good enough.

> [snip]
> But then I'm not sure this is a very good analogy--the workflow for making Clinic
> changes is very different from people hacking on Lib/*.py.

Agreed.

On Fri, Aug 27, 2021 at 10:06 PM Guido van Rossum
<report@bugs.python.org> wrote:
> [snip]
> FWIW in my attempts to time this, it looks like the perf benefits of Eric's approach are
> close to those of deep-freezing. And deep-freezing causes much more bloat of the
> source code and of the resulting binary.

The question of freeze vs deep-freeze (i.e. is deep-freeze better
enough) is one we can discuss separately, and your point here is
probably the fundamental center of that discussion.  However, I don't
think it has a lot of bearing on the change proposed in this issue.

> [snip]
> I think the only solution here was hinted at in the python-dev thread from 2018: have
> a command-line flag to turn it on or off (e.g. -X deepfreeze=1/0) and have a policy for
> what the default for that flag should be (e.g. on by default in production builds, off by
> default in developer builds -- anything that doesn't use --enable-optimizations).

Agreed.

> [snip]
> it wasn't so clear that code objects should be immutable -- that realization came later,
> when Greg Stein proposed making them ROM-able. That didn't work out, but the
> notion that code objects should be strictly mutable (to the python user, at least)
> was born

This sounds like an interesting story.  Do you have any mailing list
links handy?  (Otherwise I can search the archives.)

> In fact, Eric's approach freezes everything in the encodings package, which turns out
> to be a lot of files and a lot of code (lots of simple data tables expressed in code), and
> I found that for basic startup time, it's best not to deep-freeze the encodings module
> except for __init__.py, aliases.py and utf_8.py.

Yeah, this is something to consider.  FWIW, in my testing, dropping
encodings.* from the
list of frozen modules reduced the performance gains (from 20 ms to 21 ms).

-eric

On Fri, Aug 27, 2021 at 11:14 PM Larry Hastings <report@bugs.python.org> wrote:
> [snip] On the other hand: if we made a viable tool that could consume some arbitrary
> set of .py files and produce a C file, and said C file could then be compiled into a
> shared library, end users could enjoy this speedup over the subset of the standard
> library their program used, and perhaps even their own source tree(s).

Yeah, that would be interesting to investigate.

On Sat, Aug 28, 2021 at 5:17 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> Eric's approach, as I understand it, is pretty much what PyRun does.
> [further details]

It's reassuring to hear that the approach is known to be viable. :)

> In fact, you save quite a bit of disk space compared to a full Python installation and
> additionally benefit from the memory mapping the OS does for sharing access to the
> marshal'ed byte code between processes.

That's a good point.

> That said, some things don't work with such an approach, e.g. a few packages
> include additional data files which they expect to find on disk. Since those are
> not available anymore, they fail.
>
> For PyRun I have patched some of those packages to include the data in form of
> Python modules instead, so that it gets frozen as well, e.g. the Python grammar files.

For stdlib modules it wouldn't be a big problem to set __file__ on
frozen modules.
Would that be enough to solve the problem?

On Sat, Aug 28, 2021 at 5:41 PM Gregory Szorc <report@bugs.python.org> wrote:
> When I investigated freezing the standard library for PyOxidizer, I ran into a rash
> of problems. The frozen importer doesn't behave like PathFinder. It doesn't
> (didn't?) set some common module-level attributes

This is mostly fixable for stdlib modules.  Which attributes would
need to be added?  Are there other missing behaviors?

> Also, when I last looked at the CPython source, the frozen importer performed
> a linear scan of its indexed C array performing strcmp() on each entry until it
> found what it was looking for. So adding hundreds of modules could result in
> sufficient overhead and justify using a more efficient lookup algorithm.
> (PyOxidizer uses Rust's HashMap to index modules by name.)

Yeah, we noticed this too.  I wasn't sure it was something to worry
about at first because we're not freezing the entire stdlib.  We're
freezing on the order of 10, plus all the (80+) encoding modules.  I
figured we could look at an alternative to that linear search
afterward if it made sense.

> * Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems.

We'll do what we already do with importlib: run the tests against both
the frozen and the source modules.  Thanks for the reminder to do this
though!

On Sat, Aug 28, 2021 at 5:53 PM Gregory Szorc <report@bugs.python.org> wrote:
> Oh, PyOxidizer also ran into more general issues with the frozen importer in that
> it broke various importlib APIs. e.g. because the frozen importer only supports
> bytecode, you can't use .__loader__.get_source() to obtain the source of a module.
> This makes tracebacks more opaque and breaks legitimate API consumers relying
> on these importlib interfaces.

Good point.  Supporting more of the FileLoader API on the frozen
loader is something to look into, at least for stdlib modules.

> The fundamental limitations with the frozen importer are why I implemented my
> own meta path importer (implemented in pure Rust), which is more fully featured,
> like the PathFinder importer that most people rely on today. That importer is
> available on PyPI (https://pypi.org/project/oxidized-importer/) and has its own API
> to facilitate PyOxidizer-like functionality
> (https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer.html) if anyone
> wants to experiment with it.

Awesome!  I'll take a look.

On Sat, Aug 28, 2021 at 6:14 PM Guido van Rossum <report@bugs.python.org> wrote:
> I agree that we should shore up the frozen importer -- probably in a separate PR though.
> (@Eric: do you think this is worth its own bpo issue?)

Yeah.

-eric

At this point, here are the open questions I'm seeing:

+ The editing-stdlib-.py-files problem:
   * use a command-line flag to opt-out of frozen modules?
   * use a build flag to opt out (e.g. a configure flag or a new Py_NO_FROZEN or even Py_DEBUG)?
   * ignore frozen modules if it's a dev build?
   * (note: importlib._bootstrap and importlib._bootstrap_external must always be frozen, regardless of a flag)
   * accommodate users of an installed Python that sometimes edit stdlib modules while debugging code?
   * always emit a warning if a frozen module is ignored (in favor of the source module)?

+ Compatibility:
   * set __file__ (and __path__) on frozen modules?
   * store the hash of the source file for frozen modules (in the frozen .h files)?
   * support more of the FileLoader API on the frozen loader?

+ Penalty for too many frozen modules:
   * should we only freeze the UTF-8 encoding?
   * should we use something other than linear search for looking up frozen modules?

FWIW, I think the ideal mechanism for a dev build will be to opt in to using frozen modules (instead of the source modules).  Otherwise it is too easy for the unaware contributor to waste substantial time figuring out why their changes are not getting used.

Consequently, here's my order of preference for ignoring frozen modules:

1. use Py_DEBUG as an opt-out flag
   (if we think contributors are editing stdlib modules on a build without Py_DEBUG then that isn't good enough)
2. automatically skip frozen modules if it's a dev build, with an explicit configure flag to opt in to frozen modules

> * tricks to inject hooks ASAP (e.g. coverage.py swaps the encodings module) may lose their entry point

FWIW, I asked Ned Batchelder about this and he said this approach ("fullcoverage" [1]) was added to support running coverage on the stdlib.  It doesn't affect other users of coverage.py.  He didn't have any info on where this is used currently, though I'm pretty sure we do run coverage in CI.  Furthermore, the devguide talks about running coverage.py on the stdlib and initially indicates that modules imported during startup are not covered. [2]  However, it does have a section talking about using "fullcoverage" to cover startup modules.

I expect that any solution we make for contributors editing stdlib source files will resolve the potential issue for coverage.py.


[1] https://github.com/nedbat/coveragepy/tree/master/coverage/fullcoverage
[2] https://devguide.python.org/coverage/?highlight=coverage#measuring-coverage
[3] https://devguide.python.org/coverage/?highlight=coverage#coverage-results-for-modules-imported-early-on

> For stdlib modules it wouldn't be a big problem to set __file__ on
> frozen modules.
> Would that be enough to solve the problem?

What do you set __file__ to? Do you still materialize the .py[c] files on disk and set to that? (This would make the most sense.) If you support setting __file__ on a frozen module, how does this work at the C level? A stdlib module would want a different __file__ from a 3rd party module using the frozen module API. This would seemingly require enhancements to the frozen modules C API or some kind of hackery only available to stdlib frozen modules.

>> When I investigated freezing the standard library for PyOxidizer, I ran into a rash
>> of problems. The frozen importer doesn't behave like PathFinder. It doesn't
>> (didn't?) set some common module-level attributes
> This is mostly fixable for stdlib modules.  Which attributes would
> need to be added?  Are there other missing behaviors?

It was a few years ago and I can't recall specifics. I just remember that after encountering a couple unrelated limitations with the frozen importer I decided it was far too primitive to work as a general importer and decided I'd have to roll my own.

> Good point.  Supporting more of the FileLoader API on the frozen
> loader is something to look into, at least for stdlib modules.

>> I agree that we should shore up the frozen importer -- probably in a separate PR though.
>> (@Eric: do you think this is worth its own bpo issue?)
> Yeah.

I have some observations about the implications of this. I typed up a long comment but then realized someone would probably complain about me distracting from the technical parts of this issue. Which forum is most appropriate for this less technical commentary? (I want to lay out the case for building out an official importer far more advanced than frozen importer.)

> FWIW, I asked Ned Batchelder about this and he said this approach ("fullcoverage" [1]) was added to support running coverage on the stdlib. [...]

The docs you pointed out in [3] (where it talks about a *horrible hack you should never use" :-) should be amended with something explaining that "you need to comment out the line "<encodings.*>" from frozen_module.py for this to work, else the frozen version of the encodings module will take priority over the imposter from "fullcoverage"."

[Gregory Szorc]


> What do you set __file__ to? [...]

Exactly. I think it should not be set, just like it's not set for builtin modules.

> I have some observations about the implications of this. I typed up a long comment but then realized someone would probably complain about me distracting from the technical parts of this issue. Which forum is most appropriate for this less technical commentary? (I want to lay out the case for building out an official importer far more advanced than frozen importer.)

That seems to be something for python-dev. (There used to be an "import-sig" but the last mention I have from it is about that list shutting down for lack of traffic.) Or if you're still looking for more brainstorming you could try python-ideas first.

> At this point, here are the open questions I'm seeing:

> + The editing-stdlib-.py-files problem: [...]

> + Compatibility: [...]

? + Penalty for too many frozen modules: [...]

> FWIW, I think the ideal mechanism for a dev build will be to opt in to using frozen modules (instead of the source modules).  Otherwise it is too easy for the unaware contributor to waste substantial time figuring out why their changes are not getting used.

Agreed. I don't care much about people (even myself) editing installed modules. But I care a lot about people who do a git checkout and build from source editing modules and testing them without doing an install (Personally I never install what I build from source except to test the installation process.)

> Consequently, here's my order of preference for ignoring frozen modules:

> 1. use Py_DEBUG as an opt-out flag
>    (if we think contributors are editing stdlib modules on a build without Py_DEBUG then that isn't good enough)
> 2. automatically skip frozen modules if it's a dev build, with an explicit configure flag to opt in to frozen modules.

I propose to only opt in by default in **PGO builds**. After all what we're doing is another extreme optimization.

It should always be possible to opt in using some -X flag (e.g. to debug the freeze import loader) and it should also always be possible to opt *out* (for those cases where you want to edit an installed stdlib module in-place in anger).

I don't know how the -X mechanism works exactly but probably we could make the spelling

  python -X freeze=on|off

with a dynamic default.

FWIW, I'd be okay with doing the -X flag in a separate PR.

On Mon, Aug 30, 2021 at 2:22 PM Guido van Rossum <report@bugs.python.org> wrote:
> I propose to only opt in by default in **PGO builds**. After all what we're doing is another extreme optimization.
>
> It should always be possible to opt in using some -X flag (e.g. to debug the freeze import loader) and it should also always be possible to opt *out* (for those cases where you want to edit an installed stdlib module in-place in anger).
>
> I don't know how the -X mechanism works exactly but probably we could make the spelling
>
>   python -X freeze=on|off
>
> with a dynamic default.

+1 to all that

> set __file__ (and __path__) on frozen modules?

See https://bugs.python.org/issue21736

On 31.08.2021 20:14, Brett Cannon wrote:
> 
> Brett Cannon <brett@python.org> added the comment:
> 
>> set __file__ (and __path__) on frozen modules?
> 
> See https://bugs.python.org/issue21736

The patch on that ticket is straight from PyRun, where the
__file__ location is set in a way which signals that the file
does not exist, but instead is baked into the executable:

>>> import os
>>> os.__file__
'<pyrun>/os.py'

Not doing this breaks too many tests in the test suite for no
good reason, which is why I mentioned "practicality beats
purity" in the ticket.

I don't want all frozen header files to be committed in git repository.
Can't we just build them during build process?

On Tue, Aug 31, 2021 at 10:05 PM Inada Naoki <report@bugs.python.org> wrote:
> I don't want all frozen header files to be committed in git repository.
> Can't we just build them during build process?

That's a good point (and an interesting one).  Only two of the frozen
modules are necessary (_frozen_importlib_external and
_frozen_importlib_external, to bootstrap the import system).  So for
those two it makes sense to have them in the git repository.

For all the rest it isn't necessary.  The only advantage is that
contributors don't have to think about them and they will be
guaranteed to be there.  However, if someone clones the repo they have
to build Python, so the frozen modules will get created anyway at that
point.

So I'm fine with not committing all those modules.  This will require
that all those files be added to the .gitignore file.  (I'll update my
PR accordingly.)

-eric

On Tue, Aug 31, 2021 at 12:14 PM Brett Cannon <report@bugs.python.org> wrote:
> > set __file__ (and __path__) on frozen modules?
>
> See https://bugs.python.org/issue21736

Great!  I'll take a look.

For the module metadata problem: one potential approach to that for "designed to be frozen" stdlib modules is to set the values directly in the module code, rather than trying to set them automatically in the frozen import machinery.

It should also be possible to delete the implicitly created metadata fields and use module level dynamic attribute retrieval to find the stdlib source code for tracebacks and introspection purposes without incurring any start up costs.

FWIW, I've not found the importer for frozen modules to be lacking
features. When using frozen modules, you don't expect to see source
code, so the whole part about finding source code is not really
relevant for that use case.

The only lacking part I found regarding frozen modules is support
for these in pkgutil.py. But that's easy to add:

--- /home/lemburg/egenix/projects/PyRun/Python-3.8.0/Lib/pkgutil.py
2019-10-14 1>
+++ ./Lib/pkgutil.py    2019-11-17 11:36:38.404752218 +0100
@@ -315,20 +315,27 @@
         return self.etc[2]==imp.PKG_DIRECTORY

     def get_code(self, fullname=None):
+        # eGenix PyRun needs pkgutil to also work for frozen modules,
+        # since pkgutil is used by the runpy module, which is needed
+        # to implement the -m command line switch.
+        if self.code is not None:
+            return self.code
         fullname = self._fix_name(fullname)
-        if self.code is None:
-            mod_type = self.etc[2]
-            if mod_type==imp.PY_SOURCE:
-                source = self.get_source(fullname)
-                self.code = compile(source, self.filename, 'exec')
-            elif mod_type==imp.PY_COMPILED:
-                self._reopen()
-                try:
-                    self.code = read_code(self.file)
-                finally:
-                    self.file.close()
-            elif mod_type==imp.PKG_DIRECTORY:
-                self.code = self._get_delegate().get_code()
+        mod_type = self.etc[2]
+        if mod_type == imp.PY_FROZEN:
+            self.code = imp.get_frozen_object(fullname)
+            return self.code
+        elif mod_type==imp.PY_SOURCE:
+            source = self.get_source(fullname)
+            self.code = compile(source, self.filename, 'exec')
+        elif mod_type==imp.PY_COMPILED:
+            self._reopen()
+            try:
+                self.code = read_code(self.file)
+            finally:
+                self.file.close()
+        elif mod_type==imp.PKG_DIRECTORY:
+            self.code = self._get_delegate().get_code()
         return self.code

     def get_source(self, fullname=None):

If you reduce the number of modules being frozen you could probably manage
to land this (or most of it) before tackling those other issues.

On Mon, Sep 13, 2021 at 2:59 PM Guido van Rossum <report@bugs.python.org> wrote:
> If you reduce the number of modules being frozen you could probably manage
> to land this (or most of it) before tackling those other issues.

Yeah, that's what I'm doing. :)

New changeset a65c86889e208dddb26a7ebe7840c24edbcca775 by Eric Snow in branch 'main':
bpo-45020: Add -X frozen_modules=[on|off] to explicitly control use of frozen modules. (gh-28320)
https://github.com/python/cpython/commit/a65c86889e208dddb26a7ebe7840c24edbcca775

New changeset 369bf949ccbb689cd4638b29b4c0c12db79b927c by Terry Jan Reedy in branch 'main':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/369bf949ccbb689cd4638b29b4c0c12db79b927c

New changeset 8a9396cf1d9e1ce558841095e1ce0d3c23b7a8aa by Miss Islington (bot) in branch '3.10':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/8a9396cf1d9e1ce558841095e1ce0d3c23b7a8aa

New changeset f71b86e0ae194613d235086755c6a44266978be1 by Miss Islington (bot) in branch '3.9':
bpo-45020: Don't test IDLE with frozen module. (GH-28344)
https://github.com/python/cpython/commit/f71b86e0ae194613d235086755c6a44266978be1

New changeset cbeb81971057d6c382f45ecce92df2b204d4106a by Eric Snow in branch 'main':
bpo-45020: Freeze some of the modules imported during startup. (gh-28335)
https://github.com/python/cpython/commit/cbeb81971057d6c382f45ecce92df2b204d4106a

At this point the fundamental work is done.  Here are some follow-up tasks to wrap up this issue:

* freeze the remaining stdlib modules imported during startup (os, site, codecs, encodings.*)
   + blocked by bpo-45186 and bpo-45188
* default to "on" (except if actually running out of the source tree)
   + blocked by bpo-45211 (if we want to minimize disk access)
* always default to "on" if it's a PGO build, even if running out of the source tree
* stop tracking the frozen module .h files in the repo
   + blocked by bpo-45188
* (maybe) freeze modules imported for "python -m ..." (e.g. runpy)
* (maybe) freeze a small subset of the encodings (e.g. UTF-8)
   + blocked by a valid __path__ (see below) and consequently bpo-45211

Other related follow-up tasks:

* (maybe) make frozen stdlib modules more like source modules, with __file__ and __path__ (see bpo-21736)
   + blocked by bpo-45211
   + __path__ allows us to have frozen stdlib packages with non-frozen submodules (e.g. encodings)
* (maybe) freeze other modules (e.g. commonly used modules, sysconfig)
* (maybe) use something other than linear search for looking up frozen modules (see bpo-45213)

I would move "default to "on" (except if actually running out of the source tree)" to the "maybe" category. I left a few comments in other deps. I think we should start by turning this on by default in PGO builds.

Separately, I encourage you to collect reliable performance numbers. It would be nice to see a dip on speed.python.org for this benchmark:

https://speed.python.org/timeline/#/?exe=12&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on

(but that won't show up until we turn this on by default for PGO builds).

On Wed, Sep 15, 2021 at 12:03 PM Guido van Rossum
<report@bugs.python.org> wrote:
> I would move "default to "on" (except if actually running out of the source tree)" to the "maybe" category. I left a few comments in other deps. I think we should start by turning this on by default in PGO builds.

Sounds good.

> Separately, I encourage you to collect reliable performance numbers. It would be nice to see a dip on speed.python.org for this benchmark:

I'll do that.

New changeset a9757bf34d8b4cb3c24bbb70d50a06c815e2e8f3 by Eric Snow in branch 'main':
bpo-45020: Drop the frozen .h files from the repo. (gh-28375)
https://github.com/python/cpython/commit/a9757bf34d8b4cb3c24bbb70d50a06c815e2e8f3

Looks like that last commit broke one of the buildbots:

https://buildbot.python.org/all/#/builders/483/builds/812

I'll fix that right away.

FWIW, Python/frozen_modules/__hello__.h is still in the repo somehow.

Never mind.  It's pretty late here so I'm going to revert it and sort it out in the morning.

New changeset 9fd87a5fe5c468cf94265365091267838b004b7f by Eric Snow in branch 'main':
bpo-45020: Revert "Drop the frozen .h files from the repo." (gh-28380)
https://github.com/python/cpython/commit/9fd87a5fe5c468cf94265365091267838b004b7f

Is bpo-45220 "Windows builds sometimes fail on Azure and GitHub Action: fatal error RC1116: RC terminating after preprocessor errors" related to this issue?

I left a comment on bpo-45220.

New changeset fdc6b3d9316501d2f0068a1bf4334debc1949e62 by Eric Snow in branch 'main':
bpo-45020: Drop the frozen .h files from the repo. (gh-28392)
https://github.com/python/cpython/commit/fdc6b3d9316501d2f0068a1bf4334debc1949e62

It's no longer possible to build Python from a different directory. Example:

git clean -fdx
mkdir build
cd build/
../configure --with-pydebug
make

It fails with:

../Programs/_freeze_module importlib._bootstrap ../Lib/importlib/_bootstrap.py ../Python/frozen_modules/importlib__bootstrap.h
make: ../Programs/_freeze_module: No such file or directory

I'm working on a fix.

There is also a minor issue, "make regen-frozen" changes the end of line of two PCbuild/ files: see bpo-45231.

New changeset 41551ee7e24fb6c58846836d3655dbb212281206 by Victor Stinner in branch 'main':
bpo-45020: Fix build out of source tree (GH-28410)
https://github.com/python/cpython/commit/41551ee7e24fb6c58846836d3655dbb212281206

Thanks for fixing that, Victor!

(Ah, now I recall how the $(srcdir) mechanism works again. Generated files do *not* have a $(srcdir) prefix, since they are generated in the destination directory. Hence the fix for the buildbot failure a few days ago. I take back my skepticism about that.)

BTW, why does the script still run Programs/_freeze_module over all the modules to be frozen? Isn't that up to the Makefile or its Windows equivalent?

New changeset 090591636c4f03ce06a039079bd7716a5b23631e by Eric Snow in branch 'main':
bpo-45020: Freeze os, site, and codecs. (gh-28398)
https://github.com/python/cpython/commit/090591636c4f03ce06a039079bd7716a5b23631e

On Fri, Sep 17, 2021 at 4:22 PM Guido van Rossum <report@bugs.python.org> wrote:
> BTW, why does the script still run Programs/_freeze_module over all the modules to be frozen? Isn't that up to the Makefile or its Windows equivalent?

Yeah, it used to matter but we probably don't need to do that any longer.

It would be nice to freeze argparse.py and its dependencies.  For command-line tools, startup time is important.

> It would be nice to freeze argparse.py and its dependencies.  For command-line tools, startup time is important.

I quickly checked, and argparse has at least these dependencies:

argparse
re
enum
types
operator
functools
collections
keyword
reprlib
sre_compile
sre_parse
sre_constants
copyreg
gettext

Raymond, do you think we should also freeze the dependencies of runpy (so "python -m <module>" also starts faster)?

That would be

runpy
importlib
warnings
importlib.machinery
importlib.util
importlib._abc
contextlib
collections
keyword
operator
reprlib
functools
types

(I didn't dedupe this from the previous list.)

With all these modules frozen we'd probably run more risk of two things:

- frozen modules don't have a __file__ (and are missing a few other, less commonly used, attributes)
- editing a frozen module requires rebuilding, or running with -X frozen_modules=off

> Raymond, do you think we should also freeze the dependencies
> of runpy (so "python -m <module>" also starts faster)?

Yes, please.  

The '-m' load option can be considered core startup functionality.  It is often the recommended way to launch command line tools outside of a virtual environment.

   python -m pip install some_package

runpy startup time matters a lot for "python3 -m module" startup  time!

In Python 3.10 (bpo-41006 and bpo-41718), I reduced the number of modules imported by runpy:

"The runpy module now imports fewer modules. The python3 -m module-name command startup time is 1.4x faster in average. On Linux, python3 -I -m module-name imports 69 modules on Python 3.9, whereas it only imports 51 modules (-18) on Python 3.10. (Contributed by Victor Stinner in bpo-41006 and bpo-41718.)"

https://docs.python.org/dev/whatsnew/3.10.html#optimizations

--

For argparse, maybe it could use a few lazy imports to reduce the number of indirect impots on "import argparse"?

I'm still torn about the need for a __file__ attribute. Assuming it's more likely that people write code that *reads* stdlib source for some reason (maybe printing more context for error messages) than *writing* it (which would be truly strange), the amount of 3rd party code that might break due to the lack of __file__ can be expected to be larger than the amount of code (or number of users) that would be confused to having a __file__ pointing to a file that isn't actually read by the interpreter.

Okay, so I'm no longer torn. We should set __file__ for frozen modules based on the pre-computed location of the stdlib.

What about if there isn't a pre-computed location for __file__? I could imagine a self-contained CPython build where there is no concept of a file location on disk for anything using this.

On 22.09.2021 20:47, Brett Cannon wrote:
> What about if there isn't a pre-computed location for __file__? I could imagine a self-contained CPython build where there is no concept of a file location on disk for anything using this.

This does work and is enough to make most code out there happy.

I use e.g. "<pyrun>/os.py" in PyRun. There is no os.py file to load,
but tracebacks and inspection tools work just fine with this.

I'll throw out that __file__ is strictly optional, per https://docs.python.org/3/reference/datamodel.html. IMO it is more important to define importlib.abc.InspectLoader than __file__ (so tracebacks have source context).

Given that __file__ is optional, I'll throw out the idea of shipping modules without __file__/__cached__ and see what issues/pushback occurs. I anticipate this will likely result in having to restore advertising __file__ in frozen modules in 3.11 due to popular demand. But we /could/ use this as an opportunity to start attempting to ween people off __file__ and onto more appropriate APIs, like importlib.resources. (importlib.resources was introduced in 3.7 and I'm not sure that's old enough to force people onto without relying on a shim like https://pypi.org/project/importlib-resources/.)

Alternatively, defining a path hook referencing the python executable that knows how to service frozen modules could also work. zipimporter does this (__file__ is <zip_path>/<virtual_module_path>) and it is surprisingly not as brittle as one would think. It might be a good middle ground.

New changeset 7c801e0fa603b155eab3fd19698aa90854ac5a7b by Eric Snow in branch 'main':
bpo-45020: Fix some corner cases for frozen module generation. (gh-28538)
https://github.com/python/cpython/commit/7c801e0fa603b155eab3fd19698aa90854ac5a7b

Eric, I noticed that you are freezing os.py. Please be aware that
this module is often being used as indicator for where the stdlib
was installed (the stdlib itself does this in site.py to read the
LICENSE and the test suite also uses os.__file__ in a couple of
places).

It may be worth changing the stdlib to pick a different module
as landmark for this purpose.

Also: Unless you have added the .__file__ attribute to frozen
modules, much of this landmark checking code will fail... which is
the reason I added the attribute to frozen modules in PyRun.

The plan is to add __file__ back, based on where the stdlib lives (we won’t do any stat() calls for frozen files).

Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c and PC/getpathp.c.

On 25.09.2021 18:20, STINNER Victor wrote:
> 
> STINNER Victor <vstinner@python.org> added the comment:
> 
> Marc-Andre: I suppose that you're talking about LANDMARK in Modules/getpath.c and PC/getpathp.c.

Now that you mention it: yes, that as well :-) But os.py is used in the
Python stdlib code as well, just search for "os.__file__" to see a few
such uses.

If you search for ".__file__" you'll find that there are quite a few
cases in the test suite expecting that attribute on other stdlib modules
as well. The attribute may officially be optional, but in reality a
lot of code expects to find it.

New changeset 45ca1c04139300ec0289a32f78c7ac922a4f7b07 by Eric Snow in branch 'main':
bpo-45020: Do not freeze <pkg>/__init__.py twice. (gh-28635)
https://github.com/python/cpython/commit/45ca1c04139300ec0289a32f78c7ac922a4f7b07

New changeset 7e5c107541726b90d3f2e6e69ef37180cf58335d by Eric Snow in branch 'main':
bpo-45020: Add more test cases for frozen modules. (gh-28664)
https://github.com/python/cpython/commit/7e5c107541726b90d3f2e6e69ef37180cf58335d

@santhu_reddy12, why did you assign this to the Parser category? IMO this issue is clearly in the Build category. (We haven't met, I assume you have triage permissions?)

And it's most definitely 3.11, not 3.9. (Did you mean to change a different issue?)

New changeset 08285d563e64c179a56ab2f952345b3dbcdb54f3 by Eric Snow in branch 'main':
bpo-45020: Identify which frozen modules are actually aliases. (gh-28655)
https://github.com/python/cpython/commit/08285d563e64c179a56ab2f952345b3dbcdb54f3

Whoa. os.path is not always an alias for posixpath, is it?

Nope.  On Windows, os.path is "ntpath".

On Tue, Oct 5, 2021 at 11:31 AM Guido van Rossum <report@bugs.python.org> wrote:
> Whoa. os.path is not always an alias for posixpath, is it?

Steve brought this to my attention a couple weeks ago.  Bottom line:
the frozen module entry is only there for checks, not for actual
import, but should probably be removed regardless.  See
https://bugs.python.org/issue45272.

New changeset b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b by Eric Snow in branch 'main':
bpo-45020: Default to using frozen modules unless running from source tree. (gh-28940)
https://github.com/python/cpython/commit/b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b

could changes related to this be the cause of https://bugs.python.org/issue45506 ?

out of tree builds in main usually cannot pass key tests today.  they often hang or blow up with strange exceptions.

Is gh-28940 only for UNIX?

I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

I also put a printf() in is_dev_env() and found that it returns 0 on this check:

    /* If dirname() is the same for both then it is a dev build. */
    if (len != _Py_find_basename(stdlib)) {
        return 0;
    }

I assume that's because the binary (in my case at least) is at PCbuild\amd64\python.exe which is not the same as my current directory (which is the repo root).

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <report@bugs.python.org> wrote:
> Is gh-28940 only for UNIX?

It wasn't meant to be. :(

> I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

I'll look into this.

New changeset 6afb285ff0790471a6858e44f85d143f07fda70c by Eric Snow in branch 'main':
bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997)
https://github.com/python/cpython/commit/6afb285ff0790471a6858e44f85d143f07fda70c

On Mon, Oct 18, 2021 at 7:14 PM Guido van Rossum <report@bugs.python.org> wrote:
> Is gh-28940 only for UNIX?
> I built on Windows with default options (PCbuild\build.bat) and it looks like the frozen modules are used by default even though I am running in the source directory. (I put a printf() call in unmarshal_frozen_code().)

FYI, I opened https://bugs.python.org/issue45651 for sorting this out.

I consider this done.  There is some lingering follow-up work, for which I've created a number of issues:

* https://bugs.python.org/issue45396 - -X frozen_modules=off ignores custom frozen modules
* https://bugs.python.org/issue45395 - custom modules override frozen stdlib modules
* https://bugs.python.org/issue45651 - frozen_modules default not right on Windows
* https://bugs.python.org/issue45272 - "os.path" should be in the list of frozen modules
* https://bugs.python.org/issue45653 - freeze encodings
* https://bugs.python.org/issue45654 - freeze runpy
* https://bugs.python.org/issue45660 - freeze argparse
* https://bugs.python.org/issue45661 - freeze other stdlib modules
* https://bugs.python.org/issue45273 - os-specific modules frozen unnecessarily?
* https://bugs.python.org/issue45096 - update Tools/freeze
* https://bugs.python.org/issue45379 - improving frozen-related errors
* https://github.com/python/devguide/issues/756 - add devguide info about frozen modules

New changeset 1cbaa505d007e11c4a1f0d2073d72b6c02c7147c by Guido van Rossum in branch 'main':
bpo-45696: Deep-freeze selected modules (GH-29118)
https://github.com/python/cpython/commit/1cbaa505d007e11c4a1f0d2073d72b6c02c7147c

New changeset 5c4b19ec49a5fbad65a682225f7cfed8b78f2a2f by Christian Heimes in branch 'main':
bpo-45020: Fix strict-prototypes warning (GH-29755)
https://github.com/python/cpython/commit/5c4b19ec49a5fbad65a682225f7cfed8b78f2a2f