Title: [C API] Add getter functions for PyFrameObject and maybe move PyFrameObject to the internal C API
Type: Stage: patch review
Components: C API Versions: Python 3.9
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, nedbat, scoder, vstinner
Priority: normal Keywords: patch

Created on 2020-04-28 13:17 by vstinner, last changed 2021-11-11 02:07 by nedbat.

Pull Requests
URL Status Linked Edit
PR 19755 merged vstinner, 2020-04-28 14:07
PR 19756 merged vstinner, 2020-04-28 14:45
PR 19757 merged vstinner, 2020-04-28 15:27
PR 19764 closed vstinner, 2020-04-28 17:27
PR 19765 merged vstinner, 2020-04-28 17:33
PR 23598 merged vstinner, 2020-12-01 15:10
PR 23801 merged vstinner, 2020-12-16 14:15
Messages (14)
msg367527 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 13:17
Similarly to bpo-39573 (make PyObject opaque) and bpo-39947 (make PyThreadState opaque), I propose to add getter functions to access PyFrameObject members without exposing the PyFrameObject structure in the C API.

The first step is to identify common usage of the PyFrameObject structure inside CPython code base to add getter functions, and maybe a few setter functions as well.

frameobject.h is not part of Python.h, but it's part of the public C API. The long term plan is to move PyFrameObject structure to the internal C API to hide implementation details from the public C API.
msg367532 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 14:32
New changeset 7c59d7c9860cdbaf4a9c26c9142aebd3259d046e by Victor Stinner in branch 'master':
bpo-40421: Add pyframe.h header file (GH-19755)
msg367535 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 15:07
New changeset b8f704d2190125a7750b50cd9b67267b9c20fd43 by Victor Stinner in branch 'master':
bpo-40421: Add Include/cpython/code.h header file (GH-19756)
msg367545 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 17:01
New changeset a42ca74fa30227e2f89a619332557cf093a937d5 by Victor Stinner in branch 'master':
bpo-40421: Add PyFrame_GetCode() function (GH-19757)
msg367550 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 17:45
I looked how Cython uses PyFrameObject:

* read f_lasti
* read/write f_back
* write f_lineno
* read f_localsplus
* read/write f_trace


* Cython/Debugger/ code using the Python API of gdb to read PyFrameObject.f_lasti. It it used to compute the line number of a frame. The python_step() function puts a watch point on "f->f_lasti".
* Cython/Utility/Coroutine.c: set PyFrameObject.f_back using "f->f_back = tstate->frame;" and "Py_CLEAR(f->f_back);".
* Cython/Utility/ModuleSetupCode.c, __Pyx_PyFrame_SetLineNumber(): set PyFrameObject.f_lineno member. The limited C API flavor of this function does nothing, since this member cannot be set in the limited C API.
* Cython/Utility/ObjectHandling.c, __Pyx_PyFrame_GetLocalsplus(): complex implementation to access PyFrameObject.f_localsplus:

  // Initialised by module init code.
  static size_t __pyx_pyframe_localsplus_offset = 0;

  #include "frameobject.h"
  // This is the long runtime version of
  //     #define __Pyx_PyFrame_GetLocalsplus(frame)  ((frame)->f_localsplus)
  // offsetof(PyFrameObject, f_localsplus) differs between regular C-Python and Stackless Python.
  // Therefore the offset is computed at run time from PyFrame_type.tp_basicsize. That is feasible,
  // because f_localsplus is the last field of PyFrameObject (checked by Py_BUILD_ASSERT_EXPR below).
  #define __Pxy_PyFrame_Initialize_Offsets()  \
    ((void)__Pyx_BUILD_ASSERT_EXPR(sizeof(PyFrameObject) == offsetof(PyFrameObject, f_localsplus) + Py_MEMBER_SIZE(PyFrameObject, f_localsplus)), \
     (void)(__pyx_pyframe_localsplus_offset = ((size_t)PyFrame_Type.tp_basicsize) - Py_MEMBER_SIZE(PyFrameObject, f_localsplus)))
  #define __Pyx_PyFrame_GetLocalsplus(frame)  \
    (assert(__pyx_pyframe_localsplus_offset), (PyObject **)(((char *)(frame)) + __pyx_pyframe_localsplus_offset))

* Cython/Utility/Profile.c, __Pyx_TraceLine(): read PyFrameObject.f_trace.

* Cython/Utility/Profile.c, __Pyx_TraceSetupAndCall(): set PyFrameObject.f_trace.
msg367604 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-29 01:28
New changeset 703647732359200c54f1d2e695cc3a06b9a96c9a by Victor Stinner in branch 'master':
bpo-40421: Add PyFrame_GetBack() function (GH-19765)
msg368342 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2020-05-07 14:13
"maybe a few setter functions as well"
Please don't add any setter functions, that would a major change to the VM and would need a PEP.

Also, could you remove PyFrame_GetLastInstr(PyFrameObject *frame)?
The only purpose of `f_lasti` is to get the line number and that can be done directly via `PyFrame_GetLineNumber(PyFrameObject *frame)`

Perhaps Stefan can tell us why Cython needs to access the internals of the frame object.
msg368350 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-07 15:13
> Please don't add any setter functions, that would a major change to the VM and would need a PEP.

There is no such "major change". PyFrameObject structure was fully exposed in the public C API since the earliest Python version.

I don't see how adding setter is a major change, since it's already possible to directly modify *any* field of PyFrameObject without any kind of protection.

My plan is to have two milestones:

A) Make the structure opaque, so it's not longer possible to directly access it.

B) Deprecate or remove a few getter or setter functions, or move them to the internal C API.

I don't think that moving directly to step (B) is a realistic migration plan. So far, nobody analyzed all C extensions on PyPI to see how PyFrameObject is accessed.

I prefer to move slowly, step by step.

Breaking C extensions which currently *modify* directly PyFrameObject on purpose doesn't seem like a reasonable option to me.


I'm trying to distinguish functions which should be "safe"/"portable" from the ones which may be too "CPython specific":

* I added Include/pyframe.h which is included by Include/Python.h for "safe"/"portable" functions:

  * PyFrame_GetLineNumber()
  * PyFrame_GetCode()

* All other functions are added to Include/cpython/frameobject.h which is excluded from the limited C API:

  * PyFrame_GetBack()

Note: Compared to accessing directly PyFrameObject.f_code, PyFrame_GetCode() also avoids the issue of borrowed references since it returns a strong reference.

PyFrame_GetBack() looks specific to the current implementation of CPython. Another implementation might decide to not chain frames. Or maybe don't provide an easy way to traverse the chain of frames. Or at least, it might be a different API than PyFrame_GetBack().


> Also, could you remove PyFrame_GetLastInstr(PyFrameObject *frame)?

I didn't add it :-) It's the pending PR 19764.

I didn't merge it since it's unclear to me if it's a good idea to directly expose it or not. Cython uses it, but Cython also abuses CPython internals in general, usually for best performances :-)

*Maybe* one compromise would be to add a private _PyFrame_GetLastInstr() to ease migration to step (A) (replace direct access to the structure with function calls).
msg368424 - (view) Author: Stefan Behnel (scoder) * (Python committer) Date: 2020-05-08 06:51
Adding to the list above:

"f_lasti" is used in "", which is an almost exact copy of the one in CPython's "Tools/gdb/" for debugging Python code in gdb. If that implementation changes, we can probably adapt, especially since it uses runtime generated code. I don't find a C function for this necessary at this point, given that CPython will also need to access its own internals here in some way.

"f_localsplus" is used in the FASTCALL implementation and seems specific to Py3.6/7(?) which lacks the vectorcall protocol. So I guess it won't impact 3.8+ anymore. (The reason why its access is so complex is that StacklessPython has a slightly different frame struct, but we need to be binary(!) compatible with both.)

"f_back" is used in the coroutine implementation for correct exception traceback building, and the usage is pretty much copied from CPython. I doubt that there is much use for setting it outside of "code that tries to reimplement CPython behaviour", but Cython has to do exactly that here.

"f_lineno" is also used for traceback generation, because Cython creates frames when an exception occurs and needs to tell it what the current code line is. It's only used on newly created frames, although I can't guarantee that that will never change. Being able to read and set it seems reasonable.

"f_trace" is obviously used for tracing/profiling, and there isn't currently a good C-API for that, besides talking to frame struct fields (which is quite efficient when it comes to performance).
msg383167 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-16 14:08
New changeset fcc6935384b933fbe1a1ef659ed455a3b74c849a by Victor Stinner in branch 'master':
Add symbols of the stable ABI to python3dll.c (GH-23598)
msg383209 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-16 21:41
New changeset 166286849048eccadecf02b242dbc4042b780944 by Victor Stinner in branch '3.9':
Add symbols of the stable ABI to python3dll.c (GH-23598) (GH-23801)
msg403814 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-13 10:04
The coverage project has a ctrace C extension which access PyFrameObject.f_lasti which is gone in Python 3.11. It uses MyFrame_lasti() helper to handle Python 3.10 lasti change:
// The f_lasti field changed meaning in 3.10.0a7. It had been bytes, but
// now is instructions, so we need to adjust it to use it as a byte index.
#if PY_VERSION_HEX >= 0x030A00A7
#define MyFrame_lasti(f)    (f->f_lasti * 2)
#define MyFrame_lasti(f)    f->f_lasti
#endif // 3.10.0a7

f_lasti is used for two things in coverage/ctracer/tracer.c:

(1) get the last opcode:
            /* Need to distinguish between RETURN_VALUE and YIELD_VALUE. Read
             * the current bytecode to see what it is.  In unusual circumstances
             * (Cython code), co_code can be the empty string, so range-check
             * f_lasti before reading the byte.
            int bytecode = RETURN_VALUE;
            PyObject * pCode = MyFrame_GetCode(frame)->co_code;
            int lasti = MyFrame_lasti(frame);

            if (lasti < PyBytes_GET_SIZE(pCode)) {
                bytecode = PyBytes_AS_STRING(pCode)[lasti];
            if (bytecode != YIELD_VALUE) {
                int first = MyFrame_GetCode(frame)->co_firstlineno;
                if (CTracer_record_pair(self, self->pcur_entry->last_line, -first) < 0) {
                    goto error;

(2) get the line number, with a special case for generator which is not started yet (lasti < 0)
    /* A call event is really a "start frame" event, and can happen for
     * re-entering a generator also.  f_lasti is -1 for a true call, and a
     * real byte offset for a generator re-entry.
    if (frame->f_lasti < 0) {
        self->pcur_entry->last_line = -MyFrame_GetCode(frame)->co_firstlineno;
    else {
        self->pcur_entry->last_line = PyFrame_GetLineNumber(frame);

Since Python 3.10.0a3, PyFrame_GetLineNumber() handles the case of negative f_lasti, thanks to the commit 877df851c3ecdb55306840e247596e7b7805a60a related to the PEP 626 implementation:
PyCode_Addr2Line(PyCodeObject *co, int addrq)
    if (addrq < 0) {
        return co->co_firstlineno;

=> coverage would need an abstraction to get the last opcode: use case (1).

I recall that an old version of asyncio also had to get latest opcode, in pure Python, to workaround the CPython bpo-21209 bug:

+# Check for CPython issue #21209
+def has_yield_from_bug():
+    class MyGen:
+        def __init__(self):
+            self.send_args = None
+        def __iter__(self):
+            return self
+        def __next__(self):
+            return 42
+        def send(self, *what):
+            self.send_args = what
+            return None
+    def yield_from_gen(gen):
+        yield from gen
+    value = (1, 2, 3)
+    gen = MyGen()
+    coro = yield_from_gen(gen)
+    next(coro)
+    coro.send(value)
+    return gen.send_args != (value,)
+_YIELD_FROM_BUG = has_yield_from_bug()
+del has_yield_from_bug


+    if _YIELD_FROM_BUG:
+        # For for CPython issue #21209: using "yield from" and a custom
+        # generator, generator.send(tuple) unpacks the tuple instead of passing
+        # the tuple unchanged. Check if the caller is a generator using "yield
+        # from" to decide if the parameter should be unpacked or not.
+        def send(self, *value):
+            frame = sys._getframe()
+            caller = frame.f_back
+            assert caller.f_lasti >= 0
+            if caller.f_code.co_code[caller.f_lasti] != _YIELD_FROM:
+                value = value[0]
+            return self.gen.send(value)
+    else:
+        def send(self, value):
+            return self.gen.send(value)

Hopefully, this code could be be removed from asyncio, since the bug was fixed (and asyncio is now only maintained in the Python stdlib, it's not longer a third party module).
msg403815 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-13 10:12
See also bpo-45247: [C API] Add explicit support for Cython to the C API.
msg406144 - (view) Author: Ned Batchelder (nedbat) * (Python triager) Date: 2021-11-11 02:07
I went ahead and changed the code to this:

#if PY_VERSION_HEX >= 0x030B00A0
// 3.11 moved f_lasti into an internal structure. This is totally the wrong way
// to make this work, but it's all I've got until
// is resolved.
#include <internal/pycore_frame.h>
#define MyFrame_lasti(f)    ((f)->f_frame->f_lasti * 2)
#elif PY_VERSION_HEX >= 0x030A00A7
// The f_lasti field changed meaning in 3.10.0a7. It had been bytes, but
// now is instructions, so we need to adjust it to use it as a byte index.
#define MyFrame_lasti(f)    ((f)->f_lasti * 2)
#define MyFrame_lasti(f)    ((f)->f_lasti)

If we had an API that let me distinguish between call and resume and between return and yield, I could get rid of this.
Date User Action Args
2021-11-11 02:07:51nedbatsetnosy: + nedbat
messages: + msg406144
2021-10-13 10:12:39vstinnersetmessages: + msg403815
2021-10-13 10:04:56vstinnersetmessages: + msg403814
2020-12-16 21:41:55vstinnersetmessages: + msg383209
2020-12-16 14:15:17vstinnersetpull_requests: + pull_request22659
2020-12-16 14:08:32vstinnersetmessages: + msg383167
2020-12-01 15:10:09vstinnersetpull_requests: + pull_request22464
2020-05-08 06:51:12scodersetmessages: + msg368424
2020-05-07 15:13:33vstinnersetmessages: + msg368350
2020-05-07 14:13:15Mark.Shannonsetnosy: + Mark.Shannon, scoder
messages: + msg368342
2020-04-29 01:28:50vstinnersetmessages: + msg367604
2020-04-28 17:45:57vstinnersetmessages: + msg367550
2020-04-28 17:33:15vstinnersetpull_requests: + pull_request19086
2020-04-28 17:27:58vstinnersetpull_requests: + pull_request19085
2020-04-28 17:01:39vstinnersetmessages: + msg367545
2020-04-28 15:27:56vstinnersetpull_requests: + pull_request19079
2020-04-28 15:07:16vstinnersetmessages: + msg367535
2020-04-28 14:45:35vstinnersetpull_requests: + pull_request19077
2020-04-28 14:32:52vstinnersetmessages: + msg367532
2020-04-28 14:07:09vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request19076
2020-04-28 13:17:50vstinnercreate