This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: [C API] Get "self" args or non-null co_varnames from frame object with C-API
Type: enhancement Stage: patch review
Components: C API Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Henry Schreiner, Mark.Shannon, Skylion007, eric.snow, gregory.p.smith, pablogsal, terry.reedy, vstinner
Priority: high Keywords: 3.11regression, patch

Created on 2021-12-23 19:33 by Skylion007, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31198 merged vstinner, 2022-02-07 16:19
Messages (15)
msg409100 - (view) Author: Aaron Gokaslan (Skylion007) Date: 2021-12-23 19:33
Hello, I am a maintainer with the PyBind11 project. We have been following the 3.11 development branch and have noticed an issue we are encountering with changes to the C-API. 

Particularly, we have an edge case in our overloading dispatch mechanism that we used to solve by inspecting the "self" argument in the co_varnames member of the python frame object: ( However, in the new struct, the co_varnames object can now be null. There also doesn't appear to be any public API to populate it on the C-API side. Accessing it via the "inspect" module still works, but that requires us to run a Python code snippit in a potentially very hot code path: (

As such, we were hoping that either there is some new API change we have missed, or if there is some way other modern (and hopefully somewhat  stable way to access the API) so we can emulate the old behavior with the C-API.
msg409162 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-12-24 23:09
Pablo, Mark: I am guessing that at least one of you know about this.
msg411902 - (view) Author: Henry Schreiner (Henry Schreiner) * Date: 2022-01-27 17:07
It would be great to get this looked at before things start getting too locked in for 3.11!
msg411906 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2022-01-27 17:46
Yes, we should expose the tuple of variable names, both in Python and in the C-API.

Would something like
`PyCodeObject_GetVariableName()` and `PyCodeObject_GetVariableKind()` work?

In the meantime, since you were reading `co_varnames` directly, why not read `co_localsplusnames` directly? 

OOI, how do you cope with non-local self?
msg411908 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2022-01-27 17:53
In addition to what Mark said, note that co_varnames get's populated lazily by the Python-level getter for code.co_varnames.  So could you call the Python function before entering the hot path?

Regardless, a dedicated C-API for this like Mark suggested would be the better solution.
msg412206 - (view) Author: Aaron Gokaslan (Skylion007) Date: 2022-01-31 15:51
We didn't want to read colocalsplus directly because we were worried about the stability of that approach and the code complexity / readability. Also, I wasn't aware that colocalsplus would work or if that was lazily populated as well. 

The functions used in CPython to extract the args from colocalsplus do not seem to be public and would need to be reimplemented by PyBind11, right? That seems very brittle as try to support future Python versions and may break in the future.

Having a somewhat stable C-API to query this information seems like it would be the best solution, but I am open to suggestions on how to best proceed. How would you all recommend PyBind11 proceed with supporting 3.11 if not a C-API addition? The PyBind11 authors want to resolve this before the API becomes too locked down for 3.11.
msg412208 - (view) Author: Aaron Gokaslan (Skylion007) Date: 2022-01-31 16:17
`PyCodeObject_GetVariableName()` and `PyCodeObject_GetVariableKind()` work?

- Some public-gettters such as these functions would be ideal.

OOI, how do you cope with non-local self? 

- We only care about checking self to prevent an infinite recursion in our method dispatch code so I am not sure a non-local self would be applicable in this case? Correct me if I am wrong.
msg412210 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-01-31 16:51
It would be nice to have a PyFrame_GetVariable(frame, "self") function: get the value of the "frame" variable of the specified frame object.
msg412528 - (view) Author: Aaron Gokaslan (Skylion007) Date: 2022-02-04 19:58
I saw the latest Python 3.11 5A release notes on the frame API changes. Do the notes mean the only officially supported way of accessing co_varnames is now through the Python interface and the inspect module? By using PyObject_GetAttrString?

Also, the documentation in the WhatsNew is a bit unclear as PyObject_GetAttrString(frame, "f_locals") doesn't work for PyFrameObject*, only PyObject* and it doesn't describe how to get the PyObject* version of FrameObject. The same problem also happens when trying to access the co_varnames field of the PyCodeObject*.
msg412547 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-04 22:26
> PyObject_GetAttrString(frame, "f_locals") doesn't work for PyFrameObject*

Oh, would you mind to elaborate?

Example in Python:

$ ./python
Python 3.11.0a5+ 
>>> import sys
>>> f=sys._getframe()
>>> f.f_locals
{'__name__': '__main__', '__doc__': None, ...}
msg412761 - (view) Author: Aaron Gokaslan (Skylion007) Date: 2022-02-07 16:02
The frame object I am referring to was:

PyFrameObject *frame = PyThreadState_GetFrame(PyThreadState_Get());

This frame can not be used with PyObject_GetAttrString. Is there anyway to get the PyObject* associated with a PyFrameObject*? It seems weird that some functionality is just not accessible using the Stable ABI of PyThreadState_GetFrame . 

To elabroate: I was referring to the migration guide in the changelog btw:

    f_code: removed, use PyFrame_GetCode() instead. Warning: the function returns a strong reference, need to call Py_DECREF().

    f_back: changed (see below), use PyFrame_GetBack().

    f_builtins: removed, use PyObject_GetAttrString(frame, "f_builtins").
    // this frame object actually has to be a PyObject*, the old one was a  PyFrameObject* . Dropping this in does not work. 
    f_globals: removed, use PyObject_GetAttrString(frame, "f_globals").

    f_locals: removed, use PyObject_GetAttrString(frame, "f_locals").

    f_lasti: removed, use PyObject_GetAttrString(frame, "f_lasti").

I tried importing sys._getframe(), but that gave an attribute error interestingly enough. Run a full code snippit here works:, but is really slow and we would like avoid having to rely on it. Not to mention relying on a function that is an starts with an underscore seems like it really should be avoided.
msg412762 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-07 16:15
Example of C code that I added to _testcapi:
static PyObject *
get_caller_locals(PyObject *self, PyObject *Py_UNUSED(args))
    PyFrameObject *frame = PyThreadState_GetFrame(PyThreadState_Get());
    if (frame == NULL) {
    return PyObject_GetAttrString(frame, "f_locals");

Python example:
import _testcapi

def f():
    x = 1
    y = 2


Output on Python 3.11:
{'x': 1, 'y': 2}

=> it just works.

A PyFrameObject is a regular Python object, you can use functions like PyObject_GetAttrString().

Maybe I missed something, correct me if I'm wrong.
msg412763 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-07 16:17
> Is there anyway to get the PyObject* associated with a PyFrameObject*?

Ah. I see. If you pass a PyFrameObject* frame to PyObject_GetAttrString(), you get a compiler warning.

You should cast it explicitly: PyObject_GetAttrString((PyObject*)frame, "f_locals").
msg412764 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-07 16:19
I proposed GH-31198 to fix the compiler warnings in the doc.
msg412768 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-02-07 16:46
New changeset a89772c79183e3e62bf61b92077a04f6ebcc4a2b by Victor Stinner in branch 'main':
bpo-46166: Fix compiler warnings in What's New in Python 3.11 (GH-31198)
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90324
2022-02-07 16:46:31vstinnersetmessages: + msg412768
2022-02-07 16:20:35vstinnersettitle: Get "self" args or non-null co_varnames from frame object with C-API -> [C API] Get "self" args or non-null co_varnames from frame object with C-API
2022-02-07 16:19:53vstinnersetmessages: + msg412764
2022-02-07 16:19:23vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request29369
2022-02-07 16:17:08vstinnersetmessages: + msg412763
2022-02-07 16:15:32vstinnersetmessages: + msg412762
2022-02-07 16:02:06Skylion007setmessages: + msg412761
2022-02-04 22:26:05vstinnersetmessages: + msg412547
2022-02-04 19:58:45Skylion007setmessages: + msg412528
2022-01-31 16:51:02vstinnersetmessages: + msg412210
2022-01-31 16:17:45Skylion007setmessages: + msg412208
2022-01-31 15:51:18Skylion007setmessages: + msg412206
2022-01-27 19:06:04gregory.p.smithsetkeywords: + 3.11regression
priority: normal -> high

nosy: + gregory.p.smith
2022-01-27 17:53:31eric.snowsetmessages: + msg411908
2022-01-27 17:46:21Mark.Shannonsetmessages: + msg411906
2022-01-27 17:07:58Henry Schreinersetmessages: + msg411902
2022-01-06 12:53:56Mark.Shannonsetnosy: + eric.snow
2021-12-27 12:57:28corona10setnosy: + vstinner
2021-12-24 23:09:38terry.reedysetnosy: + Mark.Shannon, pablogsal, terry.reedy
messages: + msg409162
2021-12-23 19:51:59Henry Schreinersetnosy: + Henry Schreiner
2021-12-23 19:33:30Skylion007create