classification
Title: [C API] Add explicit support for Cython to the C API
Type: enhancement Stage:
Components: C API Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, scoder, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2021-09-20 10:59 by Mark.Shannon, last changed 2021-10-14 10:52 by vstinner.

Messages (6)
msg402224 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-09-20 10:59
As the C API has evolved it has grown features in an ad-hoc way, driven by the needs to whoever has bothered to add the code.

Maybe we should be a bit more principled about this. Specifically we should make sure that there is a well defined interface between CPython and the other major components of the Python ecosystem.
The obvious places to start are with Cython and Numpy.

This issue deals specifically with Cython. I will leave it to someone who know more about Numpy to open an issue for Numpy.

Matching Cython issue: https://github.com/cython/cython/issues/4382


This issue is an attempt to stop the annual occurrence of bugs like https://bugs.python.org/issue43760#msg393401
msg403812 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-13 09:38
We need to add C API to abstract access to structures used by Cython:

* bpo-39947: PyThreadState
* bpo-40421: PyFrameObject
msg403819 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2021-10-13 10:27
I disagree. All that is doing is locking in the current poor interface.

We do need to extend the C API for these uses, yes. But we need an API that addresses Cython's needs directly and at a higher level.

For example, Cython often wants to insert a frame into the call stack for debugging and introspection.

Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

E.g. something like:

int PushNativeFrame(const char *filename);
void PopNativeFrame(void);
int CurrentNativeFrame_SetLineNumber(int lineno);

We might even want to use such an API ourselves in some modules.
msg403821 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-13 10:30
> Rather than have Cython laboriously create a Python frame with fake code object, etc., it would make much more sense for us to offer an API to insert a "native" frame into the stack and update the line number.

I agree with that.
msg403823 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-10-13 11:21
I agree with Mark. Instead of exposing internal details in low-level API we should add more high-level API to satisfy concrete needs. It will give us more freedom of changing internals in future.
msg403894 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-14 10:52
My first attempt: Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing()
https://github.com/python/cpython/pull/28542

It adds an abstraction on accesses to PyThreadState.tracing and PyThreadState.cframe.use_tracing members.
History
Date User Action Args
2021-10-14 10:52:54vstinnersetmessages: + msg403894
2021-10-13 11:21:08serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg403823
2021-10-13 10:30:18vstinnersetmessages: + msg403821
2021-10-13 10:27:05Mark.Shannonsetmessages: + msg403819
2021-10-13 10:12:30vstinnersettitle: Add explicit support for Cython to the C API. -> [C API] Add explicit support for Cython to the C API
2021-10-13 09:38:46vstinnersetnosy: + vstinner
messages: + msg403812
2021-09-20 10:59:47Mark.Shannoncreate