Python 3.8 moved PyInterpreterState to the internal C API (commit be3b295838547bba267eb08434b418ef0df87ee0 of bpo-35886)... which caused bpo-38500 issue.

In Python 3.9, I provided Py_EnterRecursiveCall() and Py_LeaveRecursiveCall() as regular functions for the limited API: commit f4b1e3d7c64985f5d5b00f6cc9a1c146bbbfd613 of bpo-38644. Previously, there were defined as macros, but these macros didn’t compile with the limited C API which cannot access PyThreadState.recursion_depth field (the structure is opaque in the limited C API).

That's an enhancement for the limited C API, but PyThreadState is still exposed to the "cpython" C API (Include/cpython/).


We should prepare the C API to make the PyThreadState structure opaque. It cannot be done at once, there are different consumers of the PyThreadState structure. In CPython code base, I found:

* Py_TRASHCAN_BEGIN_CONDITION and Py_TRASHCAN_END macros access tstate->trash_delete_nesting field. Maybe we can hide these implementation details into new private function calls.

* faulthandler.c: faulthandler_py_enable() reads tstate->interp. We should maybe provide a getter function.

* _tracemalloc.c: traceback_get_frames() reads tstate->frame. We should maybe provide a getter function.

* Private _Py_EnterRecursiveCall() and _Py_LeaveRecursiveCall() access tstate->recursion_depth. One solution is to move these functions to the internal C API.


faulthandler and _tracemalloc are low-level debugging C extensions. Maybe it's ok for them to use the internal C API. But they are examples of C extensions accessing directly the PyThreadState structure.


See also bpo-39946 "Is it time to remove _PyThreadState_GetFrame() hook?" about PyThreadState.frame.