classification
Title: Lingering subinterpreters should be implicitly cleared on shutdown
Type: crash Stage: needs patch
Components: Versions: Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, nanjekyejoannah, ncoghlan, petr.viktorin
Priority: normal Keywords: 3.7regression

Created on 2019-03-07 13:17 by ncoghlan, last changed 2019-03-30 12:09 by ncoghlan.

Messages (6)
msg337392 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-03-07 13:17
https://docs.python.org/3/c-api/init.html#c.Py_EndInterpreter states that "Py_FinalizeEx() will destroy all sub-interpreters that haven’t been explicitly destroyed at that point."

As discussed in https://github.com/hexchat/hexchat/issues/2237, Python 3.7+ doesn't currently do that - it calls Py_FatalError instead.

That change came from https://github.com/python/cpython/pull/1728, which was based on my initial PEP 432 refactoring work, and I didn't realise that implicitly cleaning up lingering subinterpreters was a documented behaviour.

So I think we should just fix it to behave as documented, and add a new regression test to make sure it doesn't get broken again in the future.
msg337743 - (view) Author: Joannah Nanjekye (nanjekyejoannah) * (Python triager) Date: 2019-03-12 14:24
I have been wondering where the regression to test this can be put..in test__xxsubinterpreters.py may be?
msg337852 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2019-03-13 14:47
Joannah, yes, that looks like a good place. Eric Snow might have more info; he wrote that module.

As for testing Py_FatalError, there's an assert_python_failure function in test.support.script_helper.
msg339149 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-03-29 21:02
Interestingly, I noticed this independently today. :)

Here's what I wrote in #36477 (which I've closed as a duplicate):

When using subinterpreters, any that exist when Py_FinalizeEx() is called do not appear to get cleaned up during runtime finalization.  Maybe I've been looking at the code too much and I'm missing something. :)

This really isn't a problem except for embedders that use subinterpreters (where we're leaking memory).  However, even with the "python" executable it can have an impact because the subinterpreters' non-daemon threads will exit later than expected. (see #36469 & #36476)

The solution would be to finalize all subinterpreters at the beginning of Py_FinalizeEx(), right before the call to wait_for_thread_shutdown().  This means calling Py_EndInterpreter() for all the runtime's interpreters (except the main one).  It would also mean setting a flag (_PyRuntime.interpreters.finalizing?) right before that to disallow creation of any more subinterptreters.
msg339150 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-03-29 21:07
test__xxsubinterpreters is a great place for tests that exercise use of subinterpreters, including most lifecycle operations.  There are also one or two subinterpreter-related tests in test_embed.  However, for this issue the interplay with runtime finalization means tests should probably stay with other tests that exercise Py_FinalizeEx().
msg339191 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2019-03-30 12:09
I think test_embed would be the right home for this, as there's an existing test case there for subinterpreter lifecycles and repeated init/finalize cycles: https://github.com/python/cpython/blob/ddbb978e1065dde21d1662386b26ded359f4b16e/Programs/_testembed.c#L43

The test case here would be similar, but it wouldn't need the outer loop - it would just create a handful of subinterpreters, but instead of ending each one before creating the next one the way the existing test does, what it would instead do is:

* setup as per the existing test case
* create a pair of subinterpeters, using a copy of the existing loop, but omitting the `Py_EndInterpreter` call
* switch back to the main interpreter
* create a second pair of subinterpeters
* switch back to the main interpreter
* call Py_Finalize

It also occurs to me that we don't currently have a test case for what happens if you call Py_Finalize from a subinterpreter rather than the main interpreter.
History
Date User Action Args
2019-03-30 12:09:07ncoghlansetmessages: + msg339191
2019-03-29 21:07:25eric.snowsetmessages: + msg339150
2019-03-29 21:02:51eric.snowsetmessages: + msg339149
2019-03-29 21:00:35eric.snowlinkissue36477 superseder
2019-03-13 14:47:17petr.viktorinsetmessages: + msg337852
2019-03-12 14:24:25nanjekyejoannahsetmessages: + msg337743
2019-03-07 14:17:18nanjekyejoannahsetnosy: + nanjekyejoannah
2019-03-07 13:17:43ncoghlancreate