Author indygreg
Recipients BTaskaya, Mark.Shannon, brandtbucher, brett.cannon, eric.snow, gvanrossum, indygreg, larry, lemburg, nascheme, ronaldoussoren
Date 2021-08-28.23:41:17
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1630194077.37.0.138862505783.issue45020@roundup.psfhosted.org>
In-reply-to
Content
When I investigated freezing the standard library for PyOxidizer, I ran into a rash of problems. The frozen importer doesn't behave like PathFinder. It doesn't (didn't?) set some common module-level attributes that are documented by the importer "specification" to be set and this failed a handful of tests and lead to runtime issues or breakage in 3rd party packages (such as random packages looking for a __file__ on a common stdlib module).

Also, when I last looked at the CPython source, the frozen importer performed a linear scan of its indexed C array performing strcmp() on each entry until it found what it was looking for. So adding hundreds of modules could result in sufficient overhead and justify using a more efficient lookup algorithm. (PyOxidizer uses Rust's HashMap to index modules by name.)

I fully support more aggressive usage of frozen modules in the standard library to speed up interpreter startup. However, if you want to ship this as enabled by default, from my experience with PyOxidizer, I highly recommend:

* Make sure you run unit tests against the frozen modules. If you don't do this, subtle differences in how the different importers behave will lead to problems.
* Shoring up the implementation of the frozen importer to make it better conform with the importer specification.
* Be prepared for people to complain about the loss of __file__. e.g. https://github.com/indygreg/PyOxidizer/issues/69 / https://pyoxidizer.readthedocs.io/en/stable/oxidized_importer_behavior_and_compliance.html?highlight=__file__#file-and-cached-module-attributes
History
Date User Action Args
2021-08-28 23:41:17indygregsetrecipients: + indygreg, lemburg, gvanrossum, brett.cannon, nascheme, ronaldoussoren, larry, Mark.Shannon, eric.snow, brandtbucher, BTaskaya
2021-08-28 23:41:17indygregsetmessageid: <1630194077.37.0.138862505783.issue45020@roundup.psfhosted.org>
2021-08-28 23:41:17indygreglinkissue45020 messages
2021-08-28 23:41:17indygregcreate