Issue detected in the embedding tests of pybind11, running on the latest alpha of 3.10: pybind/pybind11#2774

I have reduced the weird issue/crash to a minimal reproducer, which consistently reproduces the crash on my Linux machine:

#include <Python.h>

int main() {
    Py_InitializeEx(1);

    Py_Finalize();
    Py_InitializeEx(1);

    PyRun_SimpleString("class Widget: pass\n"
                       "class DerivedWidget(Widget):\n"
                       "    def __init__(self):\n"
                       "        super().__init__()\n");

    Py_Finalize();

    printf("Works\n");

    return 0;
}

Removing the two lines in the middle that restart the interpreter makes the example work.

I've also bisected CPython to find the issue (3.10.0a3 is fine, 3.10.0a4 is not), and arrived at #20058 (ba3d67c being the first commit that breaks the example above). But I am not entirely sure where to start debugging.

The reproducing example above consistently crashes on my local machine (SIGABRT, exit code 134):

Fatal Python error: compiler_make_closure: lookup '__class__' in DerivedWidget 5 -1
freevars of __init__: ('__class__',)

Python runtime state: initialized

Current thread 0x00007f036485b680 (most recent call first):
<no Python frame>
Aborted (core dumped)

But note that in the pybind11 tests, the underlying issue causes a different error (Python throwing a weird, seemingly unrelated exception). So something seems to be messed up in the interpreter internals, and the above example just triggers it.

Oh. In _PyUnicode_FromId(), I made the assumption that _PyRuntime is left unchanged when Py_Initialize()Py_Finalize() is called multiple times. But I was wrong, it is always reset to zero. So I wrote PR 24193 to explicitly save/restore _PyRuntime.unicode_ids.next_index value.

Using PR 24193, msg384761 example displays "Works" instead of failing with a Python fatal error.

Wow, that was fast! Thanks!

I tried this out locally, and all pybind11's tests pass now. We can try again once there's a nightly build or new alpha :-)

New changeset 44bf57a by Victor Stinner in branch 'master':
bpo-42882: _PyRuntimeState_Init() leaves unicode next_index unchanged (GH-24193)
44bf57a

Thanks for your bug report, and thanks for testing alpha versions of Python! It's now fixed.

But I didn't feel comfortable without a regression test. So I wrote PR 24198 to add an unit test on _PyUnicode_FromId() with multiple Python initializations.

New changeset 11d13e8 by Victor Stinner in branch 'master':
bpo-42882: Add test_embed.test_unicode_id_init() (GH-24198)
11d13e8

Ok, now I can safely close the issue ;-)

New changeset 196d4de by Ken Jin in branch 'master':
bpo-42882: Fix MSVC warnings in pystate.c (GH-24440)
196d4de