Issue45395
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2021-10-06 20:21 by eric.snow, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Pull Requests | |||
---|---|---|---|
URL | Status | Linked | Edit |
PR 28778 | merged | eric.snow, 2021-10-06 22:03 |
Messages (21) | |||
---|---|---|---|
msg403335 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-06 20:21 | |
The mechanism to add custom frozen modules to the Python runtime is to set PyImport_FrozenModules (see Python/frozen.c) to some new array. This means that the default frozen modules (from _PyImport_FrozenModules) are no longer used unless explicitly added to the custom array. This is unlikely to be what the user wants. It's especially problematic since it's easy to not realize this (or forget) and forgetting essential modules (like _frozen_importlib) will cause crashes. It would probably make more sense to have PyImport_FrozenModules be an array of *additional* frozen modules, defaulting to an empty array. Before going down that route we need to be sure that isn't going to break folks that are accommodating the existing behavior. |
|||
msg403339 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-06 22:04 | |
I've posted a PR that demonstrates a reasonable solution. |
|||
msg403353 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-07 07:17 | |
I'm not sure I follow, but in any case, please make sure that the freeze tool in Tools/ continues to work with the new mechanism. The freeze tool would also need to know which modules are already frozen via the new script, so that modules don't get included twice. |
|||
msg403420 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-07 14:40 | |
On Thu, Oct 7, 2021 at 1:17 AM Marc-Andre Lemburg <report@bugs.python.org> wrote: > I'm not sure I follow, but in any case, please make sure that > the freeze tool in Tools/ continues to work with the new mechanism. > > The freeze tool would also need to know which modules are already > frozen via the new script, so that modules don't get included twice. Will do. |
|||
msg403421 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-07 14:43 | |
On 07.10.2021 16:40, Eric Snow wrote: > > On Thu, Oct 7, 2021 at 1:17 AM Marc-Andre Lemburg > <report@bugs.python.org> wrote: >> I'm not sure I follow, but in any case, please make sure that >> the freeze tool in Tools/ continues to work with the new mechanism. >> >> The freeze tool would also need to know which modules are already >> frozen via the new script, so that modules don't get included twice. > > Will do. Great, thanks, Eric. -- Marc-Andre Lemburg eGenix.com |
|||
msg403946 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-14 20:56 | |
@MAL, what's the best way to make sure Tools/freeze is still working? I don't see any tests for it in the test suite. I tried running the test in Tools/freeze/test, but I can't get that to work on main (or on the 3.10 branch). |
|||
msg404006 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-15 12:11 | |
On 14.10.2021 22:56, Eric Snow wrote: > > @MAL, what's the best way to make sure Tools/freeze is still working? I don't see any tests for it in the test suite. I tried running the test in Tools/freeze/test, but I can't get that to work on main (or on the 3.10 branch). You'd have to create a frozen binary using the standard way freeze works. I have never run those tests, so don't know whether they work, but, of course, made sure that the freeze works as basis for PyRun and patched it slightly to add features we needed. One of these days, I need to refactor PyRun into a standalone project and put it on Github (it's currently integrated into our internal single repo setup). Then it'll be easier to see the changes I made. For now, I can only reference the tar file: https://www.egenix.com/products/python/PyRun/#Download https://www.egenix.com/products/python/PyRun/#Installation I can send you an updated version for Python 3.8, if there's interest. Essentially, you need to create a Python module which runs your application, then point freeze.py to it and then compile the generated .c files using the generated Makefile. |
|||
msg404059 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-15 23:31 | |
@MAL, who's maintaining Tools/freeze? I'm not aware of who's using it (other than you, of course). It looks like PyRun isn't compatible with anything newer than 3.5, so it seems like that isn't verifying that Tools/freeze still works. Neither does it have tests that run in the test suite (nor on buildbots). So could Tools/freeze have been broken for a while? I ask because I haven't been able to get it work work on the master branch (or on 3.10). |
|||
msg404070 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-16 11:01 | |
On 16.10.2021 01:31, Eric Snow wrote: > > @MAL, who's maintaining Tools/freeze? I'm not aware of who's using it (other than you, of course). It looks like PyRun isn't compatible with anything newer than 3.5, so it seems like that isn't verifying that Tools/freeze still works. Neither does it have tests that run in the test suite (nor on buildbots). > > So could Tools/freeze have been broken for a while? I ask because I haven't been able to get it work work on the master branch (or on 3.10). I don't know who maintains it, but it's been working fine up until Python 3.8, which is the last version I ported PyRun to. There have also been a couple of patches going into the freeze tool, so this is still on the radar of at least some people other than me. It's also one of the oldest tools we have in Python and dates back to the early days of Python. Guido wrote the initial version. I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues. Would that help ? |
|||
msg404112 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-16 19:20 | |
On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg <report@bugs.python.org> wrote: > I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues. > Would that help ? Yeah, that would totally help. |
|||
msg404316 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-19 16:47 | |
On 16.10.2021 21:20, Eric Snow wrote: > > On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg > <report@bugs.python.org> wrote: >> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues. >> Would that help ? > > Yeah, that would totally help. Ok, I'll start looking into this and post updates here. |
|||
msg404343 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-19 19:42 | |
On Tue, Oct 19, 2021 at 10:47 AM Marc-Andre Lemburg <report@bugs.python.org> wrote: > Ok, I'll start looking into this and post updates here. Thanks! |
|||
msg404432 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-20 12:01 | |
On 19.10.2021 18:47, Marc-Andre Lemburg wrote: > >> On Sat, Oct 16, 2021 at 5:01 AM Marc-Andre Lemburg >> <report@bugs.python.org> wrote: >>> I can try to port PyRun to 3.9 and 3.10 to see whether I run into any issues. >>> Would that help ? >> >> Yeah, that would totally help. > > Ok, I'll start looking into this and post updates here. I have PyRun mostly working with Python 3.9. Still need to add a few new C modules, but the basics work. No changes were necessary to Tools/freeze/. The PGO build complains about test_embed not working - no surprise there. I'll patch the suite to ignore the test. BTW: Why is test_embed even used for the PGO target ? |
|||
msg404462 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-20 14:25 | |
On Wed, Oct 20, 2021 at 6:01 AM Marc-Andre Lemburg <report@bugs.python.org> wrote: > I have PyRun mostly working with Python 3.9. > ... > No changes were necessary to Tools/freeze/. Great! Thanks for getting to that so quickly. Are you going to take a look at 3.10 after you're happy with 3.9? > BTW: Why is test_embed even used for the PGO target ? Perhaps I've missed something, but I'm not clear on why PGO would be a problem for test_embed. Are you talking about a specific test in test_embed? |
|||
msg404467 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-20 14:38 | |
On 20.10.2021 16:25, Eric Snow wrote: > > Eric Snow <ericsnowcurrently@gmail.com> added the comment: > > On Wed, Oct 20, 2021 at 6:01 AM Marc-Andre Lemburg > <report@bugs.python.org> wrote: >> I have PyRun mostly working with Python 3.9. >> ... >> No changes were necessary to Tools/freeze/. > > Great! Thanks for getting to that so quickly. Are you going to take > a look at 3.10 after you're happy with 3.9? Yes, 3.10 is next, once I have 3.9 ironed out. And then I'll give 3.11 a try. >> BTW: Why is test_embed even used for the PGO target ? > > Perhaps I've missed something, but I'm not clear on why PGO would be a > problem for test_embed. Are you talking about a specific test in > test_embed? Sorry, I wasn't clear. PGO is not a problem for test_embed. I just wonder why the test_embed tests are run for creating the PGO profile files. test_embed is far from being a regular work load for Python applications. Well, I guess using the test suite for PGO is questionable anyway. It's just that we don't have anything else handy to create those profiles at Python build time :-) |
|||
msg404476 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-20 15:32 | |
On Wed, Oct 20, 2021 at 8:38 AM Marc-Andre Lemburg <report@bugs.python.org> wrote: > Yes, 3.10 is next, once I have 3.9 ironed out. And then I'll give 3.11 a try. Thanks! > Sorry, I wasn't clear. PGO is not a problem for test_embed. I just > wonder why the test_embed tests are run for creating the PGO profile > files. test_embed is far from being a regular work load for Python > applications. > > Well, I guess using the test suite for PGO is questionable anyway. > It's just that we don't have anything else handy to create those > profiles at Python build time :-) Ah, that makes sense. I expect we don't do anything more complex than run the whole suite. There have been discussions in the past about alternate workloads, but clearly nothing came of that. I suppose a comprehensive suite of high-level, (relatively) long-running benchmarks that cover the community's primary Python workloads would be ideal, but we don't have anything like that (yet). I suppose this is something we could discuss elsewhere. :) |
|||
msg404698 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-21 22:41 | |
I have an initial version of PyRun for Python 3.10 running as well. This created a few more headaches in order to make it work with setuptools and some glitches which appear to be bugs in 3.10 (https://bugs.python.org/issue45563 and https://bugs.python.org/issue45562). Nothing major, though. I'll have to check my version of the freeze tool against the one in Python 3.9 and 3.10 to see whether there's anything in the core versions which could cause the tool not to work. BTW: (My) freeze.py uses this startup code as main(): int main(int argc, char **argv) { extern int Py_FrozenMain(int, char **); /* Disabled, since we want to default to non-optimized mode: */ /* Py_OptimizeFlag++; */ Py_NoSiteFlag++; /* Don't import site.py */ PyImport_FrozenModules = _PyImport_FrozenModules; return Py_FrozenMain(argc, argv); } I still have to dig through the changes you have made, but this suggests that it replaces PyImport_FrozenModules completely with its own version, so the default freeze that you are implementing gets overridden. |
|||
msg405042 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-26 14:02 | |
FYI, I figured out the problem on my end. I wasn't using an installed python. Once I did it worked fine. |
|||
msg405044 - (view) | Author: Marc-Andre Lemburg (lemburg) * | Date: 2021-10-26 14:16 | |
On 26.10.2021 16:02, Eric Snow wrote: > > FYI, I figured out the problem on my end. I wasn't using an installed python. Once I did it worked fine. Oh, you mean you tried using it directly from the source tree ? I don't think I ever tried that direct route. When building PyRun, I first install to a temporary directory and then use this to run the freeze.py tool, generate the frozen .c files and run make to have the executable built. I've pretty much finished the port to 3.10. I'll try the main version in the next couple of days. There's currently a lot of work going on for the makesetup / Setup files (https://bugs.python.org/issue45548). I'm waiting for that to stabilize. |
|||
msg405268 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-28 21:04 | |
New changeset 074fa5750640a067d9894c69378a00ceecc3b948 by Eric Snow in branch 'main': bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778) https://github.com/python/cpython/commit/074fa5750640a067d9894c69378a00ceecc3b948 |
|||
msg405278 - (view) | Author: Eric Snow (eric.snow) * | Date: 2021-10-28 21:34 | |
This is done now. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:50 | admin | set | github: 89558 |
2021-10-28 21:34:11 | eric.snow | set | status: open -> closed resolution: fixed messages: + msg405278 stage: patch review -> resolved |
2021-10-28 21:04:42 | eric.snow | set | messages: + msg405268 |
2021-10-26 14:16:16 | lemburg | set | messages: + msg405044 |
2021-10-26 14:02:09 | eric.snow | set | messages: + msg405042 |
2021-10-21 22:41:55 | lemburg | set | messages: + msg404698 |
2021-10-20 15:32:49 | eric.snow | set | messages: + msg404476 |
2021-10-20 14:38:02 | lemburg | set | messages: + msg404467 |
2021-10-20 14:25:06 | eric.snow | set | messages: + msg404462 |
2021-10-20 12:01:30 | lemburg | set | messages: + msg404432 |
2021-10-19 19:42:26 | eric.snow | set | messages: + msg404343 |
2021-10-19 16:47:16 | lemburg | set | messages: + msg404316 |
2021-10-16 19:20:49 | eric.snow | set | messages: + msg404112 |
2021-10-16 11:20:03 | corona10 | set | nosy:
+ corona10 |
2021-10-16 11:01:05 | lemburg | set | messages: + msg404070 |
2021-10-15 23:31:13 | eric.snow | set | messages: + msg404059 |
2021-10-15 12:11:18 | lemburg | set | messages: + msg404006 |
2021-10-14 20:56:57 | eric.snow | set | messages: + msg403946 |
2021-10-07 14:43:14 | lemburg | set | messages: + msg403421 |
2021-10-07 14:40:22 | eric.snow | set | messages: + msg403420 |
2021-10-07 07:17:49 | lemburg | set | messages: + msg403353 |
2021-10-06 22:04:48 | eric.snow | set | messages: + msg403339 |
2021-10-06 22:03:56 | eric.snow | set | keywords:
+ patch stage: needs patch -> patch review pull_requests: + pull_request27112 |
2021-10-06 20:21:03 | eric.snow | create |