This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Frozen stdlib modules are discarded if custom frozen modules added.
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, corona10, eric.snow, lemburg
Priority: normal Keywords: patch

Created on 2021-10-06 20:21 by eric.snow, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 28778 merged eric.snow, 2021-10-06 22:03
Messages (21)
msg403335 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-06 20:21
The mechanism to add custom frozen modules to the Python runtime is to set PyImport_FrozenModules (see Python/frozen.c) to some new array.  This means that the default frozen modules (from _PyImport_FrozenModules) are no longer used unless explicitly added to the custom array.  This is unlikely to be what the user wants.  It's especially problematic since it's easy to not realize this (or forget) and forgetting essential modules (like _frozen_importlib) will cause crashes.

It would probably make more sense to have PyImport_FrozenModules be an array of *additional* frozen modules, defaulting to an empty array.  Before going down that route we need to be sure that isn't going to break folks that are accommodating the existing behavior.
msg403339 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-06 22:04
I've posted a PR that demonstrates a reasonable solution.
msg403353 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-07 07:17
I'm not sure I follow, but in any case, please make sure that
the freeze tool in Tools/ continues to work with the new mechanism.

The freeze tool would also need to know which modules are already
frozen via the new script, so that modules don't get included twice.
msg403420 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-07 14:40
On Thu, Oct 7, 2021 at 1:17 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> I'm not sure I follow, but in any case, please make sure that
> the freeze tool in Tools/ continues to work with the new mechanism.
>
> The freeze tool would also need to know which modules are already
> frozen via the new script, so that modules don't get included twice.

Will do.
msg403421 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-07 14:43
On 07.10.2021 16:40, Eric Snow wrote:
> 
> On Thu, Oct 7, 2021 at 1:17 AM Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
>> I'm not sure I follow, but in any case, please make sure that
>> the freeze tool in Tools/ continues to work with the new mechanism.
>>
>> The freeze tool would also need to know which modules are already
>> frozen via the new script, so that modules don't get included twice.
> 
> Will do.

Great, thanks, Eric.
 --
Marc-Andre Lemburg
eGenix.com
msg403946 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-14 20:56
@MAL, what's the best way to make sure Tools/freeze is still working?  I don't see any tests for it in the test suite.  I tried running the test in Tools/freeze/test, but I can't get that to work on main (or on the 3.10 branch).
msg404006 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-15 12:11
On 14.10.2021 22:56, Eric Snow wrote:
> 
> @MAL, what's the best way to make sure Tools/freeze is still working?  I don't see any tests for it in the test suite.  I tried running the test in Tools/freeze/test, but I can't get that to work on main (or on the 3.10 branch).

You'd have to create a frozen binary using the standard way freeze
works. I have never run those tests, so don't know whether they work,
but, of course, made sure that the freeze works as basis for PyRun
and patched it slightly to add features we needed.

One of these days, I need to refactor PyRun into a standalone project
and put it on Github (it's currently integrated into our internal
single repo setup). Then it'll be easier to see the changes I made.
For now, I can only reference the tar file:

https://www.egenix.com/products/python/PyRun/#Download
https://www.egenix.com/products/python/PyRun/#Installation

I can send you an updated version for Python 3.8, if there's
interest.

Essentially, you need to create a Python module which runs your
application, then point freeze.py to it and then compile the
generated .c files using the generated Makefile.
msg404059 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-15 23:31
@MAL, who's maintaining Tools/freeze?  I'm not aware of who's using it (other than you, of course).  It looks like PyRun isn't compatible with anything newer than 3.5, so it seems like that isn't verifying that Tools/freeze still works.  Neither does it have tests that run in the test suite (nor on buildbots).

So could Tools/freeze have been broken for a while?  I ask because I haven't been able to get it work work on the master branch (or on 3.10).
msg404070 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-16 11:01
On 16.10.2021 01:31, Eric Snow wrote:
> 
> @MAL, who's maintaining Tools/freeze?  I'm not aware of who's using it (other than you, of course).  It looks like PyRun isn't compatible with anything newer than 3.5, so it seems like that isn't verifying that Tools/freeze still works.  Neither does it have tests that run in the test suite (nor on buildbots).
> 
> So could Tools/freeze have been broken for a while?  I ask because I haven't been able to get it work work on the master branch (or on 3.10).

I don't know who maintains it, but it's been working fine up until Python 3.8,
which is the last version I ported PyRun to.

There have also been a couple of patches going into the freeze tool, so this is
still on the radar of at least some people other than me. It's also one of the
oldest tools we have in Python and dates back to the early days of Python. Guido
wrote the initial version.

I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
Would that help ?
msg404112 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-16 19:20
On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
> Would that help ?

Yeah, that would totally help.
msg404316 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-19 16:47
On 16.10.2021 21:20, Eric Snow wrote:
> 
> On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
>> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
>> Would that help ?
> 
> Yeah, that would totally help.

Ok, I'll start looking into this and post updates here.
msg404343 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-19 19:42
On Tue, Oct 19, 2021 at 10:47 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> Ok, I'll start looking into this and post updates here.

Thanks!
msg404432 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-20 12:01
On 19.10.2021 18:47, Marc-Andre Lemburg wrote:
> 
>> On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg
>> <report@bugs.python.org> wrote:
>>> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues.
>>> Would that help ?
>>
>> Yeah, that would totally help.
> 
> Ok, I'll start looking into this and post updates here.

I have PyRun mostly working with Python 3.9. Still need to add a few
new C modules, but the basics work.

No changes were necessary to Tools/freeze/. The PGO build complains
about test_embed not working - no surprise there. I'll patch the suite
to ignore the test.

BTW: Why is test_embed even used for the PGO target ?
msg404462 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-20 14:25
On Wed, Oct 20, 2021 at 6:01 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> I have PyRun mostly working with Python 3.9.
> ...
> No changes were necessary to Tools/freeze/.

Great!  Thanks for getting to that so quickly.  Are you going to take
a look at 3.10 after you're happy with 3.9?

> BTW: Why is test_embed even used for the PGO target ?

Perhaps I've missed something, but I'm not clear on why PGO would be a
problem for test_embed.  Are you talking about a specific test in
test_embed?
msg404467 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-20 14:38
On 20.10.2021 16:25, Eric Snow wrote:
> 
> Eric Snow <ericsnowcurrently@gmail.com> added the comment:
> 
> On Wed, Oct 20, 2021 at 6:01 AM Marc-Andre Lemburg
> <report@bugs.python.org> wrote:
>> I have PyRun mostly working with Python 3.9.
>> ...
>> No changes were necessary to Tools/freeze/.
> 
> Great!  Thanks for getting to that so quickly.  Are you going to take
> a look at 3.10 after you're happy with 3.9?

Yes, 3.10 is next, once I have 3.9 ironed out. And then I'll give
3.11 a try.

>> BTW: Why is test_embed even used for the PGO target ?
> 
> Perhaps I've missed something, but I'm not clear on why PGO would be a
> problem for test_embed.  Are you talking about a specific test in
> test_embed?

Sorry, I wasn't clear. PGO is not a problem for test_embed. I just
wonder why the test_embed tests are run for creating the PGO profile
files. test_embed is far from being a regular work load for Python
applications.

Well, I guess using the test suite for PGO is questionable anyway.
It's just that we don't have anything else handy to create those
profiles at Python build time :-)
msg404476 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-20 15:32
On Wed, Oct 20, 2021 at 8:38 AM Marc-Andre Lemburg
<report@bugs.python.org> wrote:
> Yes, 3.10 is next, once I have 3.9 ironed out. And then I'll give 3.11 a try.

Thanks!

> Sorry, I wasn't clear. PGO is not a problem for test_embed. I just
> wonder why the test_embed tests are run for creating the PGO profile
> files. test_embed is far from being a regular work load for Python
> applications.
>
> Well, I guess using the test suite for PGO is questionable anyway.
> It's just that we don't have anything else handy to create those
> profiles at Python build time :-)

Ah, that makes sense.  I expect we don't do anything more complex than
run the whole suite.  There have been discussions in the past about
alternate workloads, but clearly nothing came of that.  I suppose a
comprehensive suite of high-level, (relatively) long-running
benchmarks that cover the community's primary Python workloads would
be ideal, but we don't have anything like that (yet).  I suppose this
is something we could discuss elsewhere. :)
msg404698 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-21 22:41
I have an initial version of PyRun for Python 3.10 running as well.
This created a few more headaches in order to make it work with
setuptools and some glitches which appear to be bugs in 3.10
(https://bugs.python.org/issue45563 and https://bugs.python.org/issue45562).
Nothing major, though.

I'll have to check my version of the freeze tool against the one
in Python 3.9 and 3.10 to see whether there's anything in the
core versions which could cause the tool not to work.

BTW: (My) freeze.py uses this startup code as main():

int
main(int argc, char **argv)
{
        extern int Py_FrozenMain(int, char **);

        /* Disabled, since we want to default to non-optimized mode: */
        /* Py_OptimizeFlag++; */
        Py_NoSiteFlag++;        /* Don't import site.py */

        PyImport_FrozenModules = _PyImport_FrozenModules;
        return Py_FrozenMain(argc, argv);
}

I still have to dig through the changes you have made, but this
suggests that it replaces PyImport_FrozenModules completely
with its own version, so the default freeze that you are
implementing gets overridden.
msg405042 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-26 14:02
FYI, I figured out the problem on my end.  I wasn't using an installed python.  Once I did it worked fine.
msg405044 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-26 14:16
On 26.10.2021 16:02, Eric Snow wrote:
> 
> FYI, I figured out the problem on my end.  I wasn't using an installed python.  Once I did it worked fine.

Oh, you mean you tried using it directly from the source tree ?
I don't think I ever tried that direct route.

When building PyRun, I first install to a temporary directory and
then use this to run the freeze.py tool, generate the frozen .c
files and run make to have the executable built.

I've pretty much finished the port to 3.10.

I'll try the main version in the next couple of days. There's currently
a lot of work going on for the makesetup / Setup files
(https://bugs.python.org/issue45548). I'm waiting for that to stabilize.
msg405268 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-28 21:04
New changeset 074fa5750640a067d9894c69378a00ceecc3b948 by Eric Snow in branch 'main':
bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778)
https://github.com/python/cpython/commit/074fa5750640a067d9894c69378a00ceecc3b948
msg405278 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2021-10-28 21:34
This is done now.
History
Date User Action Args
2022-04-11 14:59:50adminsetgithub: 89558
2021-10-28 21:34:11eric.snowsetstatus: open -> closed
resolution: fixed
messages: + msg405278

stage: patch review -> resolved
2021-10-28 21:04:42eric.snowsetmessages: + msg405268
2021-10-26 14:16:16lemburgsetmessages: + msg405044
2021-10-26 14:02:09eric.snowsetmessages: + msg405042
2021-10-21 22:41:55lemburgsetmessages: + msg404698
2021-10-20 15:32:49eric.snowsetmessages: + msg404476
2021-10-20 14:38:02lemburgsetmessages: + msg404467
2021-10-20 14:25:06eric.snowsetmessages: + msg404462
2021-10-20 12:01:30lemburgsetmessages: + msg404432
2021-10-19 19:42:26eric.snowsetmessages: + msg404343
2021-10-19 16:47:16lemburgsetmessages: + msg404316
2021-10-16 19:20:49eric.snowsetmessages: + msg404112
2021-10-16 11:20:03corona10setnosy: + corona10
2021-10-16 11:01:05lemburgsetmessages: + msg404070
2021-10-15 23:31:13eric.snowsetmessages: + msg404059
2021-10-15 12:11:18lemburgsetmessages: + msg404006
2021-10-14 20:56:57eric.snowsetmessages: + msg403946
2021-10-07 14:43:14lemburgsetmessages: + msg403421
2021-10-07 14:40:22eric.snowsetmessages: + msg403420
2021-10-07 07:17:49lemburgsetmessages: + msg403353
2021-10-06 22:04:48eric.snowsetmessages: + msg403339
2021-10-06 22:03:56eric.snowsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request27112
2021-10-06 20:21:03eric.snowcreate