New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[subinterpreters] atexit_callfuncs() crashing within Py_Finalize() when using multiple interpreters. #50780
Comments
I am seeing a crash within Py_Finalize() with Python 3.0 in mod_wsgi. It looks like the This new problem I am seeing looks like it may be linked to where the 'atexit' module is PyImport_ImportModule("atexit");
Py_Finalize(); At a guess, the problem is because in atexit_callfuncs(): module = PyState_FindModule(&atexitmodule);
if (module == NULL)
return; still returns a module for case where imported in a sub interpreter but not in main modstate = GET_ATEXIT_STATE(module);
returns NULL for modstate for the main interpreter as PyInit_atexit() had never been called The fix would appear to be to check modstate for being NULL and return. Ie., module = PyState_FindModule(&atexitmodule);
if (module == NULL)
return;
modstate = GET_ATEXIT_STATE(module);
The only thing I am uncertain about is why PyState_FindModule() would return an object. I BTW, I have marked this as for Python 3.1 as well, but haven't tested it on that. The code in For now am using the workaround in mod_wsgi. |
I strongly recommend you move to 3.1. 3.0 will not be getting any more |
As a provider of software that others use I am just making mod_wsgi usable |
I'm sorry, I wasn't clear. We are recommending that no one use Python I should have just kept my mouth shut, especially since I don't have any |
What did you mean by 'crash'? An exception or a segfault or equivalent? I believe the finalization code was reworked a bit last fall, so could you determine whether or not there is still a problem in 3.2.1? Do not bother with 3.1.4 unless you think this is a security problem. |
Segmentation fault. The original description explains the problem is dereferencing of a NULL pointer which has a tendency to invoke such behaviour. |
There seem to be two issues at play here:
Having atexit work properly with sub-interpreters would require fixing these two issues AFAICT. |
Hmm, it seems I may be mistaken about the second point. m_index is actually generated such that all modules end up in the same position in the interpreters' respective modules_by_index lists. Sorry. |
Hmm something else: currently the atexit funcs are only called when the main interpreter exits, but that doesn't really make sense: if I register a function from a sub-interpreter, why would it execute correctly from another interpreter? All kind of fun things can happen... |
New changeset eb47af6e9e22 by Antoine Pitrou in branch '3.2': New changeset a108818aaa0d by Antoine Pitrou in branch 'default': |
Here is a patch for subinterp-wise atexit functions. |
What are the intentions with respect to atexit and sub interpreters? The original report was only about ensuring that the main interpreter doesn't crash if an atexit function was registered in a sub interpreter. So, was not expecting a change to sub interpreters in submitting this report, in as much as atexit callbacks for sub interpreters are never invoked in Python 2.X. That said, for mod_wsgi I have extended sub interpreter destruction so that atexit callbacks registered in sub interpreters are called. For mod_wsgi though, sub interpreters are only destroyed on process shutdown. For the general case, a sub interpreter could be destroyed at any time during the life of the process. If one called atexit callbacks on such sub interpreter destruction, it notionally changes the meaning of atexit, which is in process exit and not really sub interpreter exit. |
Well the atexit docs say "Functions thus registered are automatically My reasoning is:
I expect uses of sub-interpreters to be quite rare apart from mod_wsgi, () note atexit functions are even called *before module globals are |
See also https://bugs.python.org/issue31901, which reached a similar conclusion to this discussion (i.e. atexit functions should be run when the subinterpreter goes away). |
In bpo-31901, I solved this problem by getiing rid of PyState_FindModule in the atexit code, using module state and PEP-489 multiphase initialization. This needed some changes in the pystate and pylifecycle code, so the specific interpreter has a reference to its own atexit module. This way, Py_EndInterpreter() is able to call the callbacks for the given interpreter. See: #4611 |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: