This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Restarting the interpreter causes UB on 3.10.0a4
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: YannickJadoul, kj, vstinner
Priority: normal Keywords: patch

Created on 2021-01-10 13:00 by YannickJadoul, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24193 merged vstinner, 2021-01-11 17:03
PR 24198 merged vstinner, 2021-01-12 09:48
PR 24440 merged kj, 2021-02-04 22:06
Messages (8)
msg384761 - (view) Author: Yannick Jadoul (YannickJadoul) * Date: 2021-01-10 13:00
Issue detected in the embedding tests of pybind11, running on the latest alpha of 3.10: https://github.com/pybind/pybind11/issues/2774

I have reduced the weird issue/crash to a minimal reproducer, which consistently reproduces the crash on my Linux machine:

```
#include <Python.h>

int main() {
    Py_InitializeEx(1);

    Py_Finalize();
    Py_InitializeEx(1);

    PyRun_SimpleString("class Widget: pass\n"
                       "class DerivedWidget(Widget):\n"
                       "    def __init__(self):\n"
                       "        super().__init__()\n");

    Py_Finalize();

    printf("Works\n");

    return 0;
}
```

Removing the two lines in the middle that restart the interpreter makes the example work.

I've also bisected CPython to find the issue (3.10.0a3 is fine, 3.10.0a4 is not), and arrived at https://github.com/python/cpython/pull/20058 (ba3d67c2fb04a7842741b1b6da5d67f22c579f33 being the first commit that breaks the example above). But I am not entirely sure where to start debugging.

The reproducing example above consistently crashes on my local machine (SIGABRT, exit code 134):

```
Fatal Python error: compiler_make_closure: lookup '__class__' in DerivedWidget 5 -1
freevars of __init__: ('__class__',)

Python runtime state: initialized

Current thread 0x00007f036485b680 (most recent call first):
<no Python frame>
Aborted (core dumped)
```

But note that in the pybind11 tests, the underlying issue causes a different error (Python throwing a weird, seemingly unrelated exception). So something seems to be messed up in the interpreter internals, and the above example just triggers it.
msg384850 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-11 17:05
Oh. In _PyUnicode_FromId(), I made the assumption that _PyRuntime is left unchanged when Py_Initialize()Py_Finalize() is called multiple times. But I was wrong, it is always reset to zero. So I wrote PR 24193 to explicitly save/restore _PyRuntime.unicode_ids.next_index value.

Using PR 24193, msg384761 example displays "Works" instead of failing with a Python fatal error.
msg384855 - (view) Author: Yannick Jadoul (YannickJadoul) * Date: 2021-01-11 18:42
Wow, that was fast! Thanks!

I tried this out locally, and all pybind11's tests pass now. We can try again once there's a nightly build or new alpha :-)
msg384903 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-12 09:30
New changeset 44bf57aca627bd11a08b12fe4e4b6a0e1d268862 by Victor Stinner in branch 'master':
bpo-42882: _PyRuntimeState_Init() leaves unicode next_index unchanged (GH-24193)
https://github.com/python/cpython/commit/44bf57aca627bd11a08b12fe4e4b6a0e1d268862
msg384904 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-12 09:49
Thanks for your bug report, and thanks for testing alpha versions of Python! It's now fixed.

But I didn't feel comfortable without a regression test. So I wrote PR 24198 to add an unit test on _PyUnicode_FromId() with multiple Python initializations.
msg384910 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-12 10:26
New changeset 11d13e83abedabba12b28773317f1a365113e7af by Victor Stinner in branch 'master':
bpo-42882: Add test_embed.test_unicode_id_init() (GH-24198)
https://github.com/python/cpython/commit/11d13e83abedabba12b28773317f1a365113e7af
msg384912 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-12 10:34
Ok, now I can safely close the issue ;-)
msg386499 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-04 22:08
New changeset 196d4deaf4810a0bba75ba537dd40f2d71a5a634 by Ken Jin in branch 'master':
bpo-42882: Fix MSVC warnings in pystate.c (GH-24440)
https://github.com/python/cpython/commit/196d4deaf4810a0bba75ba537dd40f2d71a5a634
History
Date User Action Args
2022-04-11 14:59:40adminsetgithub: 87048
2021-02-04 22:08:11vstinnersetmessages: + msg386499
2021-02-04 22:06:50kjsetnosy: + kj

pull_requests: + pull_request23250
2021-01-12 10:34:04vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg384912

stage: patch review -> resolved
2021-01-12 10:26:59vstinnersetmessages: + msg384910
2021-01-12 09:49:45vstinnersetmessages: + msg384904
2021-01-12 09:48:48vstinnersetpull_requests: + pull_request23024
2021-01-12 09:30:16vstinnersetmessages: + msg384903
2021-01-11 18:42:40YannickJadoulsetmessages: + msg384855
2021-01-11 17:05:18vstinnersetmessages: + msg384850
2021-01-11 17:03:50vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23020
2021-01-10 18:29:48vstinnersetnosy: + vstinner
2021-01-10 13:00:18YannickJadoulcreate