Author eric.snow
Recipients LewisGaul, eric.snow, maciej.szulik, mdk, nanjekyejoannah, ncoghlan, ned.deily, petr.viktorin, vstinner
Date 2020-10-22.21:48:12
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1603403292.75.0.748240354779.issue36225@roundup.psfhosted.org>
In-reply-to
Content
> I see multiple non trivial problems:
>
> * To finalize the subinterpreter A, Py_Finalize() may switch the current Python thread state from the main interpreter to a Python thread in the subintrepeter A. It can lead to new funny issues.

I'm not sure what you mean with this.  Are you talking about how another thread might be running that threadstate already?  That doesn't sounds like a problem specific to this issue but rather a problem with Py_EndInterpreter() in general.

> * A subinterpreter can be stuck for whatever reason and refuse to stop. For example, the subinterpreter A may wait for an even from subinterpreter B. If we don't run two interpreters "in parallel" (in threads), it may be stuck forever.

The wait_for_thread_shutdown() call in Py_FinalizeEx() already has a similar behavior.  In the case of one interpreter blocking another like that, how would that be possible where it isn't already a problem?

> * Python 3.10 still has weird code to allow daemon threads to continue to run after Py_Finalize() complete. Maybe we need to add such hacks for subinterpreter which fail to be finalized? For example, kill them (exit) as soon as they attempt to acquire the GIL. Search for tstate_must_exit() in Python/ceval_gil.h.

I consider the problem of daemon threads to be a separate problem.  The solution proposed here for finalization doesn't change any behavior regarding daemon threads.

> By the way, currently Py_Finalize() calls PyInterpreterState_Clear() which call object finalizers in the main thread of the main interpreter, whereas these finalizers might expect to be called from the thread which created them.

Do you have any examples?  I'm having trouble imagining such a case.

> I don't know if we must change anything else.
>
> Again, all these problems are very complex :-(

I agree.  However, automatically finalizing all other interpreters at the beginning of Py_FinalizeEx() doesn't introduce any new problems.  At the same time, it makes sure that any resources in use in other interpreters have a chance to be released and that global resources used there don't cause crashes during runtime finalization.

> The simple option, which is sadly a backward incompatible change, is to raise a fatal error in Py_Finalize() if a subinterpreter is still running.

As long as we require Py_FinalizeEx() to be called from the main interpreter, I don't see how the proposed change (in PR #17575) would be backward incompatible.

> Maybe we can start by logging a warning into stderr for now, before introducing the backward incompatible change. Maybe even only emit it when -X dev is used? https://docs.python.org/dev/library/devmode.html

I'd like to see a ResourceWarning if any other interpreters were still around.
History
Date User Action Args
2020-10-22 21:48:12eric.snowsetrecipients: + eric.snow, ncoghlan, vstinner, ned.deily, petr.viktorin, maciej.szulik, mdk, nanjekyejoannah, LewisGaul
2020-10-22 21:48:12eric.snowsetmessageid: <1603403292.75.0.748240354779.issue36225@roundup.psfhosted.org>
2020-10-22 21:48:12eric.snowlinkissue36225 messages
2020-10-22 21:48:12eric.snowcreate