Python 3.8 moved PyInterpreterState to the internal C API (commit be3b295838547bba267eb08434b418ef0df87ee0 of bpo-35886)... which caused bpo-38500 issue.

In Python 3.9, I provided Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as regular functions for the limited API: commit f4b1e3d7c64985f5d5b00f6cc9a1c146bbbfd613 of bpo-38644. Previously, there were defined as macros, but these macros didn’t compile with the limited C API which cannot access PyThreadState.recursion_depth field (the structure is opaque in the limited C API).

That's an enhancement for the limited C API, but PyThreadState is still exposed to the "cpython" C API (Include/cpython/).


We should prepare the C API to make the PyThreadState structure opaque. It cannot be done at once, there are different consumers of the PyThreadState structure. In CPython code base, I found:

* Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macros access tstate->trash_delete_nesting field. Maybe we can hide these implementation details into new private function calls.

* faulthandler.c: faulthandler_py_enable() reads tstate->interp. We should maybe provide a getter function.

* _tracemalloc.c: traceback_get_frames() reads tstate->frame. We should maybe provide a getter function.

* Private _Py_EnterRecursiveCall() and _Py_LeaveRecursiveCall() access tstate->recursion_depth. One solution is to move these functions to the internal C API.


faulthandler and _tracemalloc are low-level debugging C extensions. Maybe it's ok for them to use the internal C API. But they are examples of C extensions accessing directly the PyThreadState structure.


See also bpo-39946 "Is it time to remove _PyThreadState_GetFrame() hook?" about PyThreadState.frame.

New changeset 224481a8c988fca12f488544edd2f01c0af2a91d by Victor Stinner in branch 'master':
bpo-39947: Move Py_EnterRecursiveCall() to internal C API (GH-18972)
https://github.com/python/cpython/commit/224481a8c988fca12f488544edd2f01c0af2a91d

As a note, externally I have to use it in pydevd to set the tracing for different threads -- i.e.: https://bugs.python.org/issue35370

Will that still be possible?

> As a note, externally I have to use it in pydevd to set the tracing for different threads -- i.e.: https://bugs.python.org/issue35370 Will that still be possible?

My intent is not to prevent third-party C extension modules to modify PyThreadState, but to make the structure opaque. I mean that we should add getter and setter function for the most commonly used PyThreadState fields.

New changeset 3f2f4fefca388bc62fc2a7d07cb6ef24197c84bd by Victor Stinner in branch 'master':
bpo-39947: Move get_recursion_depth() to _testinternalcapi (GH-18974)
https://github.com/python/cpython/commit/3f2f4fefca388bc62fc2a7d07cb6ef24197c84bd

New changeset 38965ec5411da60d312b59be281f3510d58e0cf1 by Victor Stinner in branch 'master':
bpo-39947: Hide implementation detail of trashcan macros (GH-18971)
https://github.com/python/cpython/commit/38965ec5411da60d312b59be281f3510d58e0cf1

I tested to build numpy with an opaque PyThreadState. First issue, Plex gets the current interpreter using PyThreadState.interp:

    /tmp/pip-install-aq60p8w2/Cython/Cython/Plex/Scanners.c:7447:73: erreur: déréférencement d'un pointeur du type incomplet « PyThreadState » {alias « struct _ts »}
     7447 |     PY_INT64_T current_id = PyInterpreterState_GetID(PyThreadState_Get()->interp);

We should add a PyThreadState_GetInterpreter(tstate) getter. faulthandler_py_enable() would use it for example.

Maybe _PyInterpreterState_Get() can be used, but it's a private function. There are also _PyThreadState_UncheckedGet() and _PyGILState_GetInterpreterStateUnsafe() which are worse: don't check for NULL pointers.

New changeset ff4584caca04cb3da0dbd5b1e9bf67e40adf5312 by Victor Stinner in branch 'master':
bpo-39947: Use _PyInterpreterState_GET_UNSAFE() (GH-18978)
https://github.com/python/cpython/commit/ff4584caca04cb3da0dbd5b1e9bf67e40adf5312

New changeset be79373a78c0d75fc715ab64253c9b757987a848 by Victor Stinner in branch 'master':
bpo-39947: Add PyInterpreterState_Get() function (GH-18979)
https://github.com/python/cpython/commit/be79373a78c0d75fc715ab64253c9b757987a848

New changeset 8fb02b6e1942811c8d81041e7df3f5f1f4b1d410 by Victor Stinner in branch 'master':
bpo-39947: Add PyThreadState_GetInterpreter() (GH-18981)
https://github.com/python/cpython/commit/8fb02b6e1942811c8d81041e7df3f5f1f4b1d410

I added PyInterpreterState_Get() and PyThreadState_GetInterpreter() functions to get the interpreter.

New changeset fd1e1a18fa3befe5b6eeac32e0561e15c7e5164b by Victor Stinner in branch 'master':
bpo-39947: Add PyThreadState_GetFrame() function (GH-19092)
https://github.com/python/cpython/commit/fd1e1a18fa3befe5b6eeac32e0561e15c7e5164b

New changeset 3072338642156a5edfa2deef2557288fff73c22a by Victor Stinner in branch 'master':
bpo-39947: Use PyThreadState_GetFrame() (GH-19159)
https://github.com/python/cpython/commit/3072338642156a5edfa2deef2557288fff73c22a

New changeset 0e427c6d159e86f17270770cd8dc37372e3c4004 by Victor Stinner in branch 'master':
bpo-39947: Add _PyThreadState_GetDict() function (GH-19160)
https://github.com/python/cpython/commit/0e427c6d159e86f17270770cd8dc37372e3c4004

New changeset 5c3cda0d1a850a1a9b43892f48376b8876bd5863 by Victor Stinner in branch 'master':
bpo-39947: Add PyThreadState_GetID() function (GH-19163)
https://github.com/python/cpython/commit/5c3cda0d1a850a1a9b43892f48376b8876bd5863

Cython still access multiple PyThreadState members which have no getter or setter yet.

__Pyx_PyErr_ExceptionMatchesInState():

    PyObject *exc_type = tstate->curexc_type;
    ...

=> internal _PyErr_Occurred(tstate) could solve this issue: move it the public/private API? Or expose internal _PyErr_ExceptionMatches(tstate, exc)?

__Pyx_ErrRestoreInState() is a reimplementation of internal _PyErr_Restore(): get/set curexc_type, curexc_value and curexc_traceback members.

__Pyx_PyFunction_FastCallNoKw:

static PyObject* __Pyx_PyFunction_FastCallNoKw(...) {
    ...
    ++tstate->recursion_depth;
    Py_DECREF(f);
    --tstate->recursion_depth;
    return result;
}

Why not calling Py_EnterRecursiveCall/Py_LeaveRecursiveCall?

There are likely others.

I marked bpo-35949 "Move PyThreadState into Include/internal/pycore_pystate.h" as a duplicate of this issue.

bpo-35949 lists Py_ALLOW_RECURSION and Py_END_ALLOW_RECURSION which access PyThreadState.recursion_critical member directly.

> bpo-35949 lists Py_ALLOW_RECURSION and Py_END_ALLOW_RECURSION which access PyThreadState.recursion_critical member directly.

Oh, problem solved by:

commit dcc54215ac1eb4b6fab2a9ffe1abcdf3ac4bb77e
Author: Serhiy Storchaka <storchaka@gmail.com>
Date:   Mon Oct 5 12:32:00 2020 +0300

    bpo-41936. Remove macros Py_ALLOW_RECURSION/Py_END_ALLOW_RECURSION (GH-22552)

New changeset fcc6935384b933fbe1a1ef659ed455a3b74c849a by Victor Stinner in branch 'master':
Add symbols of the stable ABI to python3dll.c (GH-23598)
https://github.com/python/cpython/commit/fcc6935384b933fbe1a1ef659ed455a3b74c849a

New changeset 166286849048eccadecf02b242dbc4042b780944 by Victor Stinner in branch '3.9':
Add symbols of the stable ABI to python3dll.c (GH-23598) (GH-23801)
https://github.com/python/cpython/commit/166286849048eccadecf02b242dbc4042b780944

About setting frame local variables, see:

* bpo-42197: Disable automatic update of frame locals during tracing
* bpo-30744: Local variable assignment is broken when combined with threads + tracing + closures
* PEP 558: Defined semantics for locals()
  https://www.python.org/dev/peps/pep-0558/

New changeset db532a09990c837ec1348e6e6bd2719f5d4a8216 by Victor Stinner in branch 'main':
bpo-39947: Remove old private trashcan C API functions (GH-26869)
https://github.com/python/cpython/commit/db532a09990c837ec1348e6e6bd2719f5d4a8216

See bpo-43760 and PR 28498 for a discussion about the PyThreadState.use_tracing member.

See also bpo-45247: [C API] Add explicit support for Cython to the C API.

I would be nice to make the PyThreadState opaque in Python 3.12. IMO it's too late for Python 3.11. Hopefully, Cython should be prepared for such change. At the beginning, maybe Cython can just use the internal C API, as it does to access the internal PyFrameObject structure.