Programs/_testembed.c contains the following test used by test_embed:

static int test_audit_subinterpreter(void)
{
    Py_IgnoreEnvironmentFlag = 0;
    PySys_AddAuditHook(_audit_subinterpreter_hook, NULL);
    _testembed_Py_Initialize();

    Py_NewInterpreter(); 
    Py_NewInterpreter(); 
    Py_NewInterpreter(); 
 
    Py_Finalize(); 

    switch (_audit_subinterpreter_interpreter_count) {
        case 3: return 0;
        case 0: return -1;
        default: return _audit_subinterpreter_interpreter_count;
    }
}

When Py_Finalize() is called, the current interpreter is a subinterpreter (the 3rd interpreter), not the main interpreter.

* Is it correct to call Py_Finalize() in such case?
* Is Python supposed to magically destroy the 3 interpreters?

In bpo-38858, I'm trying to reuse the same code to initialize and finalize the "main" interpreter and subinterpreters. I had an issue with test_audit_subinterpreter() when working on the PR 17293.

I modified my PR 17293 to not expect that Py_Finalize() can only be called from the main interpreter, but actually check if the current interpreter is the main interpreter or not. It fix test_audit_subinterpreter() but again, I'm not sure what is the correct behavior?

--

Last year, we had a similar discussion about calling Py_Main() *after* Py_Initialize(). I hacked the code to make it possible because it was supported previously, even if the Py_Main() configuration is only partially applied. But I understood that Nick Coghlan would prefer to deprecate supporting to call Py_Initialize() before Py_Main().

PEP 587 added Py_RunMain() which provides a different solution to this problem:
https://docs.python.org/dev/c-api/init_config.html#c.Py_RunMain

tl;dr Py_Finalize() probably *should* only be allowed under the main interpreter.

As you know well, when the runtime starts we do it at first relative to a partially initialized main interpreter and the finish initialization with a fully operational main interpreter.  My expectation is that some of the global runtime state relies somehow on the main interpreter's state.  Consequently, finalization would then need to happen under the main interpreter.

The same idea applies to fork-without-exec, where we destroy all interpreters but the current one.  We now only allow os.fork() in the main interpreter.  So I'd expect the same treatement for finalization.

On the other hand, if the "main" interpreter isn't special after runtime init is done (i.e. a subinterpreter is effectively indistinguishable by nature) then it certainly would not matter which one finalizes (or is active during fork() for that matter).  On top of that, the effort we are making to identify the main interpreter (both in the code and in the docs) becomes unnecessary.

If we look at it from that angle, though, we do have cases where the main interpreter is still significant (regardless of the relationship between the main interpreter and the global runtime state).  At the very least, the main interpreter is solely responsible for some global resources, like signals.  So the main interpreter cannot go away until nothing else in the runtime relies on those resources.  I expect that we won't go down that route, which means finalization should happen only under the main interpreter.

Also keep in mind that this only matters relative to the C-API and I expect only embedders (not extensions) will be finalizing the runtime.  With PEP 554 the only case where you would run into problems is when running subinterpreters in daemon threads.  And we already know about problems with daemon threads during runtime finalization! :)

> * Is Python supposed to magically destroy the 3 interpreters?

Doesn't it?  Gah.  I guess I was thinking of PyOS_AfterFork_Child(), which calls _PyInterpreterState_DeleteExceptMain(). :/  In September there was a nice comment right above Py_FinalizeEx() saying that we do not clean up other interpreters or threads, but I haven't checked when it got removed.

I could swear the topic of destroy-everything-in-PyFinalize has come up before but I don't remember anything specific.  Doing so there sure makes sense though...

This issue became a blocker issue while working on bpo-39984, when I tried to move pending calls from _PyRuntimeState to PyInterpreterState.

I wrote PR 19063 to deny calling Py_FinalizeEx() from a subinterpreter.


Eric:
> On the other hand, if the "main" interpreter isn't special after runtime init is done (i.e. a subinterpreter is effectively indistinguishable by nature) then it certainly would not matter which one finalizes (or is active during fork() for that matter).  On top of that, the effort we are making to identify the main interpreter (both in the code and in the docs) becomes unnecessary.

Well. Maybe we will be able to reach this point later, but right now, there are concrete implementation issues. The main interpreter remains special. Only the main interpreter is allowed to call fork(), spawn daemon threads, handle signals, etc.

If we manage to make subinterpreters less special, we can change Py_FinalizeEx() again later.


I asked:
> Is Python supposed to magically destroy the 3 interpreters?

Currently, Py_FinalizeEx() fails with a fatal error if there are remaining subinterpreters:
---
Fatal Python error: PyInterpreterState_Delete: remaining subinterpreters
Python runtime state: finalizing (tstate=0xfaa1c0)

Abandon (core dumped)
---

IMHO it's a good behavior.

if we want, we can modify Py_FinalizeEx() to magically end subinterpters later. The opposite may be more annoying for embedders.

See also bpo-36225: "Lingering subinterpreters should be implicitly cleared on shutdown".

I changed my mind. I managed to implement bpo-39984 without needing my PR 19063, so I closed it.

See also bpo-37776: Test Py_Finalize() from a subinterpreter.

I marked bpo-37776 "[subinterpreters] Test Py_Finalize() from a subinterpreter" as a duplicate of this issue.