As the C API has evolved it has grown features in an ad-hoc way, driven by the needs to whoever has bothered to add the code.

Maybe we should be a bit more principled about this. Specifically we should make sure that there is a well defined interface between CPython and the other major components of the Python ecosystem.
The obvious places to start are with Cython and Numpy.

This issue deals specifically with Cython. I will leave it to someone who know more about Numpy to open an issue for Numpy.

Matching Cython issue: https://github.com/cython/cython/issues/4382


This issue is an attempt to stop the annual occurrence of bugs like https://bugs.python.org/issue43760#msg393401

We need to add C API to abstract access to structures used by Cython:

* bpo-39947: PyThreadState
* bpo-40421: PyFrameObject

I disagree. All that is doing is locking in the current poor interface.

We do need to extend the C API for these uses, yes. But we need an API that addresses Cython's needs directly and at a higher level.

For example, Cython often wants to insert a frame into the call stack for debugging and introspection.

Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

E.g. something like:

int PushNativeFrame(const char *filename);
void PopNativeFrame(void);
int CurrentNativeFrame_SetLineNumber(int lineno);

We might even want to use such an API ourselves in some modules.

> Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

I agree with that.

I agree with Mark. Instead of exposing internal details in low-level API we should add more high-level API to satisfy concrete needs. It will give us more freedom of changing internals in future.

My first attempt: Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
https://github.com/python/cpython/pull/28542

It adds an abstraction on accesses to PyThreadState.tracing and PyThreadState.cframe.use_tracing members.

I created bpo-46836: "[C API] Move PyFrameObject to the internal C API".