Python 3.11 made many changes in PyFrameObject and PyThreadState structures. Code which accessed directly structure members doesn't build anymore. For example, "frame->f_code" is now fails because PyFrameObject.f_code member is done. I propose to document these changes and explain how to port code.

I will write a documentation PR.

== PyFrameObject changes ==

PyFrameObject now only creates a Python frame object on demand.

• f_code: removed, use PyFrame_GetCode() instead, warning: it returns a strong reference (Py_DECREF is needed)
• f_lineno: changed, use PyFrame_GetLineNumber()
• f_back: changed, use PyFrame_GetBack()
• f_builtins: removed, get the "f_builtins" attribute in Python
• f_globals: removed, get the "f_globals" attribute in Python
• f_locals: removed, get the "f_locals" attribute in Python
• f_valuesstack: removed
• f_stackdepth: removed
• f_gen: removed
• f_lasti: removed, get the "f_lasti" attribute in Python?
• f_iblock: removed
• f_state: removed
• f_blockstack: removed
• f_localsplus: removed

Accessing f_lineno and f_back doesn't fail with a compiler error, but these members are filled lazily. If PyFrame_GetLineNumber() is not called, it can return 0 even if the frame is running and has a line number. If PyFrame_GetBack() is not called, f_back is NULL even if the frame has a next outer frame.

== PyThreadState changes ==

• frame: removed, use PyThreadState_GetFrame(), warning: it returns a strong reference (Py_DECREF is needed)
• recursion_depth: removed, use (tstate->recursion_limit - tstate->recursion_remaining) instead
• stackcheck_counter: removed
• tracing: changed, use PyThreadState_EnterTracing() and PyThreadState_LeaveTracing(), added by bpo-43760

== Notes ==

We should also explain how to get new C API functions, like PyFrame_GetCode(), on older Python, and maybe suggest to use pythoncapi_compat to get them:

https://github.com/pythoncapi/pythoncapi_compat

See also:

• bpo-39947: "[C API] Make the PyThreadState structure opaque (move it to the internal C API)"
• bpo-40421: "[C API] Add getter functions for PyFrameObject and maybe move PyFrameObject to the internal C API"
• bpo-43760: "The DISPATCH() macro is not as efficient as it could be (move PyThreadState.use_tracing)" -- add PyThreadState_EnterTracing()

f_lineno: changed, use PyFrame_GetLineNumber()

Oh I was wrong, PyFrame_GetLineNumber() code is the same in Python 3.10. It was already a bad idea to read directly the f_lineno member in Python 3.10: PyFrame_GetLineNumber() should always be called.

Are there any C programming guides we can link to, that explain API use?
I would hope that competent C programmers would know not to read or write to undocumented fields. But if they come from a Python background, that might not be obvious.

If the advice for something is "use Python", we should probably add an API function.

New changeset 0885999 by Victor Stinner in branch 'main':
bpo-46355: Document PyFrameObject and PyThreadState changes (GH-30558)
0885999

I merged my PR so the doc can be reviewed online at https://docs.python.org/dev/whatsnew/3.11.html once it will be rendered ;-)

I would hope that competent C programmers would know not to read or write to undocumented fields. But if they come from a Python background, that might not be obvious.

Tread carefully when changing decades-old API, documented or not. The docs weren't always as useful and complete as they are now.

If the advice for something is "use Python", we should probably add an API function.

Not really. If PyObject_GetAttrString doesn't add too much overhead, IMO it's fine to use it.

New changeset a4cb319 by Petr Viktorin in branch 'main':
bpo-46355: What's New: Note that PyFrameObject are private (GH-31032)
a4cb319

New changeset ba57251 by Erlend Egeberg Aasland in branch 'main':
bpo-46355: Amend What's New in Python 3.11 C API wording (GH-31288)
ba57251

I created bpo-46836: "[C API] Move PyFrameObject to the internal C API".

New changeset ec4a580 by Victor Stinner in branch 'main':
bpo-46355: Update pythoncapi_compat project URL (GH-31670)
ec4a580