This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type
Type: Stage: resolved
Components: C API Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords: patch

Created on 2020-02-06 23:07 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 18388 merged vstinner, 2020-02-06 23:16
PR 18389 merged vstinner, 2020-02-06 23:41
PR 18390 merged vstinner, 2020-02-07 00:26
PR 18391 merged vstinner, 2020-02-07 00:55
PR 18392 merged vstinner, 2020-02-07 01:26
PR 18393 merged vstinner, 2020-02-07 02:06
PR 18394 merged vstinner, 2020-02-07 02:39
PR 18398 merged vstinner, 2020-02-07 09:47
PR 18400 merged vstinner, 2020-02-07 10:20
PR 18402 merged vstinner, 2020-02-07 11:15
PR 18411 merged corona10, 2020-02-08 03:53
PR 18419 closed shihai1991, 2020-02-09 05:48
PR 18488 merged corona10, 2020-02-12 16:54
PR 18496 merged brandtbucher, 2020-02-13 01:43
PR 18507 merged corona10, 2020-02-14 01:11
PR 18508 merged corona10, 2020-02-14 01:21
PR 18521 merged corona10, 2020-02-16 17:25
PR 18601 merged petdance, 2020-02-22 04:44
PR 18789 closed petdance, 2020-03-05 05:28
PR 18798 closed petdance, 2020-03-06 05:51
PR 18799 merged petdance, 2020-03-06 05:52
PR 18804 merged vstinner, 2020-03-06 09:43
PR 18809 merged petdance, 2020-03-06 16:51
PR 19882 merged shihai1991, 2020-05-03 12:40
PR 19975 closed ZackerySpytz, 2020-05-07 10:21
PR 20290 merged corona10, 2020-05-21 15:37
PR 20391 merged corona10, 2020-05-25 17:00
PR 20429 merged vstinner, 2020-05-26 14:19
PR 20610 merged vstinner, 2020-06-03 13:00
PR 21262 closed WildCard65, 2020-07-01 16:04
PR 21433 merged vstinner, 2020-07-10 10:17
PR 23366 merged vstinner, 2020-11-18 14:02
PR 23375 merged hroncok, 2020-11-18 18:49
PR 26493 merged vstinner, 2021-06-02 23:08
PR 26550 BTaskaya, 2021-06-06 10:59
PR 26596 merged pablogsal, 2021-06-08 11:07
PR 28128 merged vstinner, 2021-09-02 15:46
Messages (96)
msg361513 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-06 23:07
Today, CPython is leaking too many implementation through its public C API. We cannot easily change the "default" C API, but we can enhance the "limited" C API (when Py_LIMITED_API macro is defined). Example of leaking implementation details: memory allocator, garbage collector, structure layouts, etc.

Making PyObject an opaque structure would allow in the long term of modify structures to implement more efficient types (ex: list specialized for small integers), and it can prepare CPython to experiment tagged pointers.

Longer rationale:

* https://pythoncapi.readthedocs.io/
* https://pythoncapi.readthedocs.io/bad_api.html
* https://pythoncapi.readthedocs.io/optimization_ideas.html

I propose to incremental evolve the existing limited C API towards opaque PyObject, by trying to reduce the risk of breakage.

We may test changes on PyQt which uses the limited C API.

Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.
msg361514 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-06 23:38
> Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

First issues that I met when I tried that:

* C code generated by Argument Clinic is incompatible the limited C API: METH_FASTCALL, _PyArg_CheckPositional(), static _PyArg_Parser, etc. are excluded from the limited C API.
* PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like:
  Py_TYPE(self)->tp_free((PyObject*)self);
* _Py_IDENTIFIER() is not part of the limited C API
msg361515 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-06 23:39
New changeset a93c51e3a8e15f1a486d11d5b55a64f3381babe0 by Victor Stinner in branch 'master':
bpo-39573: Use Py_REFCNT() macro (GH-18388)
https://github.com/python/cpython/commit/a93c51e3a8e15f1a486d11d5b55a64f3381babe0
msg361516 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 00:11
> it can prepare CPython to experiment tagged pointers

In September 2018, Neil Schemenauer did an experiment:

* https://mail.python.org/archives/list/capi-sig@python.org/thread/EGAY55ZWMF2WSEMP7VAZSFZCZ4VARU7L/
* https://github.com/nascheme/cpython/commits/tagged_int

More recent discussion on the capi-sig list:

https://mail.python.org/archives/list/capi-sig@python.org/thread/JPUNPN3AILGXOA3C2TTSLMOFNSWJE3QX/

See also my notes:
https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doable

Wikipedia article: https://en.wikipedia.org/wiki/Tagged_pointer
msg361517 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 00:19
In the limited C API, Py_REFCNT() should be converted to:

static inline Py_ssize_t _Py_REFCNT(const PyObject *ob)
{ return ob->ob_refcnt; }
#define Py_REFCNT(ob) _Py_REFCNT(_PyObject_CAST(ob))

It would enforce the usage of newly added Py_SET_REFCNT() (PR 18389) and advertise that the object is not modified (const).

That would only be the first step towards a really opaque Py_REFCNT() function.
msg361518 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 00:22
TODO: Add Py_IS_TYPE() macro:

#define Py_IS_TYPE(ob, tp) (Py_TYPE(ob) == (tp)) 

For example, replace:

    #define PyBool_Check(x) (Py_TYPE(x) == &PyBool_Type) 

with:

    #define PyBool_Check(x) Py_IS_TYPE(x, &PyBool_Type)

IMHO it makes the code more readable.

https://github.com/nascheme/cpython/commit/c156300592dc1eab234b74ed5b7cc90a020ab82b
msg361519 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 00:24
New changeset c86a11221df7e37da389f9c6ce6e47ea22dc44ff by Victor Stinner in branch 'master':
bpo-39573: Add Py_SET_REFCNT() function (GH-18389)
https://github.com/python/cpython/commit/c86a11221df7e37da389f9c6ce6e47ea22dc44ff
msg361522 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 00:53
New changeset 0d76d2bd28ac815dabae8b07240ed002ac8fce2d by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() in abstract.c (GH-18390)
https://github.com/python/cpython/commit/0d76d2bd28ac815dabae8b07240ed002ac8fce2d
msg361523 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 01:02
Py_TYPE() is commonly used to render the type name in an error message. Example:

PyErr_Format(PyExc_TypeError,
             "cannot convert '%.200s' object to bytearray",
             Py_TYPE(arg)->tp_name);

This code has multiple issues:

* It truncates type name to 200 characters: there is no Python exception, not even a marker to indicate that the string has been truncated
* It's only the short name: the qualified name (tp_qualname) would be more helpful. The best would be to generate the fully qualified name: module + qualname.
* Py_TYPE() returns a borrowed reference which is causing multiple issues: https://pythoncapi.readthedocs.io/bad_api.html#borrowed-references

In September 2018, I created bpo-34595: "PyUnicode_FromFormat(): add %T format for an object type name". But there was disagreement, so I rejected my change.

I started "bpo-34595: How to format a type name?" thread on python-dev:
* https://mail.python.org/archives/list/python-dev@python.org/thread/HKYUMTVHNBVB5LJNRMZ7TPUQKGKAERCJ/#3UAMHYG6UF4MPLXBZORHO4JVKUBRUZ53

I didn't continue this work (until now), since it wasn't my priority.
msg361526 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 01:24
New changeset a102ed7d2f0e7e05438f14d5fb72ca0358602249 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Python and Include directories (GH-18391)
https://github.com/python/cpython/commit/a102ed7d2f0e7e05438f14d5fb72ca0358602249
msg361527 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 01:42
Make PyObject an opaque structure is also a first step towards the more ambitious project "HPy" project which is fully opaque:
https://github.com/pyhandle/hpy

This API is written from scratch and currently implemented on top on the existing C API.

The following article is a nice introduction to the overall idea:
https://morepypy.blogspot.com/2019/12/hpy-kick-off-sprint-report.html

From my point of view, the long term goal would be to get better performance on PyPy and having a single API for C extension which would be efficient on all Python implementations (not only CPython).

Currently, the C API is not only a performance issue to run C extensions on PyPy. It's also an issue in CPython. Because the C API leaks too many implementation details, we cannot experiment optimizations.

See also: https://pythoncapi.readthedocs.io/rationale.html
msg361529 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 02:04
New changeset 58ac700fb09497df14d4492b6f820109490b2b88 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Objects directory (GH-18392)
https://github.com/python/cpython/commit/58ac700fb09497df14d4492b6f820109490b2b88
msg361531 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 02:37
New changeset daa9756cb6395323d6f291efe5c7d7fdc6b2e9d8 by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in Modules directory (GH-18393)
https://github.com/python/cpython/commit/daa9756cb6395323d6f291efe5c7d7fdc6b2e9d8
msg361540 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 08:17
New changeset d2ec81a8c99796b51fb8c49b77a7fe369863226f by Victor Stinner in branch 'master':
bpo-39573: Add Py_SET_TYPE() function (GH-18394)
https://github.com/python/cpython/commit/d2ec81a8c99796b51fb8c49b77a7fe369863226f
msg361549 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 09:19
To make PyObject opaque, we would have to convert Py_INCREF() and Py_DECREF() to opaque function calls. Example:

#define Py_XINCREF(op) Py_IncRef(op)
#define Py_XDECREF(op) Py_DecRef(op)

Benchmarks should be run to measure to overhead and balance the advantages and drawbacks.
msg361555 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 09:55
Would a Py_TYPE_IS() macro help code readability?

For example:
    #define Future_CheckExact(obj) (Py_TYPE(obj) == &FutureType)
would become:
    #define Future_CheckExact(obj) (Py_TYPE_IS(obj, &FutureType))

Py_TYPE_IS() would be more efficient for tagged pointers.

I'm not sure about the macro name. Neil used Py_IS_TYPE(obj, type).

Note: Py_TYPE_EQ(obj, type) name sounds confusing since the first parameter is an object, whereas the second one is a type.
msg361557 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 10:18
New changeset c65b320a95784d2b2133926921d67ac439259e9f by Victor Stinner in branch 'master':
bpo-39573: Use Py_TYPE() macro in object.c (GH-18398)
https://github.com/python/cpython/commit/c65b320a95784d2b2133926921d67ac439259e9f
msg361590 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 11:05
New changeset b10dc3e7a11fcdb97e285882eba6da92594f90f9 by Victor Stinner in branch 'master':
bpo-39573: Add Py_SET_SIZE() function (GH-18400)
https://github.com/python/cpython/commit/b10dc3e7a11fcdb97e285882eba6da92594f90f9
msg361593 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-02-07 11:31
You have merged so much PRs today. What they do?

PyObject cannot just be made an opaque structure. The user code reads and writes its fields directly and via macros. This change would break working code.

We can encourage the user code to prepare to making PyObject an opaque structure. We need to provide a stable C API for access of PyObject fields for this. Note that there is a performance penalty of using functions instead of direct access, so you should have very good reasons to do this.
msg361607 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 18:45
> You have merged so much PRs today. What they do?

I merged changes which prepares CPython code base to make PyObject opaque. I only merged changes which should have no impact on performance, but prepare the API to make the structure opaque.

Right now, Py_SET_REFNCT() stills access directly to PyObject.ob_refcnt. But it becomes possible to make Py_SET_REFNCT() an opaque function call.

Do you see any issue with the changes that I already merged? Using PGO+LTO, static inline functions should be as efficient as the previous code using Py_REFCNT() & cie macros.


> PyObject cannot just be made an opaque structure. The user code reads and writes its fields directly and via macros. This change would break working code.

I'm trying to modifying the limited C API to make it possible: all access to PyObject fields should go through macros or function calls. The question is now how which fields are accessed and how.


> We can encourage the user code to prepare to making PyObject an opaque structure. We need to provide a stable C API for access of PyObject fields for this.

For the short term, I don't plan to make PyObject opaque, so I don't plan to enforce usage of Py_TYPE(), Py_SET_REFCNT(), etc.


> Note that there is a performance penalty of using functions instead of direct access, so you should have very good reasons to do this.

Yeah, replacing Py_REFCNT() macro with an opaque function call is likely to have an impact on performance. It should be properly measure, I'm well aware of that, I already wrote it in a previous comment ;-) I don't plan to push change such right now. And I will wait for the review of my peers (like you) for such change ;-)
msg361611 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-07 22:18
New changeset 60ac6ed5579f6666130fc264d3b748ee9575e3aa by Victor Stinner in branch 'master':
bpo-39573: Use Py_SET_SIZE() function (GH-18402)
https://github.com/python/cpython/commit/60ac6ed5579f6666130fc264d3b748ee9575e3aa
msg361626 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-02-08 12:08
"static inline" functions are not opaque - as they get inlined into 3rd-party compiled code, we can't change anything they reference, and so the structure layout is still fixed and has to be visible to the user's compiler.

I'm not totally against the changes, but it's worth pointing out that you aren't achieving what the issue title claims, so it's really just code cleanliness (and/or introducing macro-users to static inline functions ;) ).
msg361631 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-08 14:28
> "static inline" functions are not opaque

I'm well aware of that :-) But once the CPython code base will stop accessing directly PyObject fields directly, it would become possible to experiment changing PyObject layout, at least testing it in CPython.

First changes are just to prepare the code base to experiment the real change. But as Serhiy pointed out, the second part will have an impact on performance and so should be carefully benchmarked to balance advantages and drawbacks, even if it's only done in the limited C API.
msg361639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-08 23:46
New changeset 7f6f7eef5206858030cbe4f80a7c04b02781cc9a by Dong-hee Na in branch 'master':
bpo-39573: Use Py_TYPE() macro in ctypes.h (GH-18411)
https://github.com/python/cpython/commit/7f6f7eef5206858030cbe4f80a7c04b02781cc9a
msg361904 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-02-12 15:36
FYI, I am working on to add Py_IS_TYPE macro. :)
msg361960 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-02-13 15:55
Hi, guys. Is there value in adding `PyNone_Check` macro?(`_PyNone_Type` is not esposed to CAPI directly, so I am not sure about it)
If the answer is 'yes', i can add it ;)
msg361961 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-13 15:58
> Hi, guys. Is there value in adding `PyNone_Check` macro?

"obj == Py_None" is a very common pattern.

You have check how it is done in HPy: https://github.com/pyhandle/hpy

See also bpo-39511: "[subinterpreters] Per-interpreter singletons (None, True, False, etc.)".
msg361963 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-02-13 16:03
>"obj == Py_None" is a very common pattern.
>You have check how it is done in HPy: https://github.com/pyhandle/hpy
>See also bpo-39511: "[subinterpreters] Per-interpreter singletons (None, >True, False, etc.)".

Thanks, I will check it.
msg361964 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-13 17:35
New changeset 968dcd9e7a4d3aa9aaa1dfca693adf60d6b71ce7 by Brandt Bucher in branch 'master':
bpo-39573: Fix bad copy-paste in Py_SET_SIZE (GH-18496)
https://github.com/python/cpython/commit/968dcd9e7a4d3aa9aaa1dfca693adf60d6b71ce7
msg361965 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-13 17:37
New changeset d905df766c367c350f20c46ccd99d4da19ed57d8 by Dong-hee Na in branch 'master':
bpo-39573: Add Py_IS_TYPE() function (GH-18488)
https://github.com/python/cpython/commit/d905df766c367c350f20c46ccd99d4da19ed57d8
msg361971 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-13 21:50
> Hi, guys.

By the way, please find another more inclusive way to say hi, see:
https://heyguys.cc/
msg361977 - (view) Author: Hai Shi (shihai1991) * (Python triager) Date: 2020-02-14 00:50
> By the way, please find another more inclusive way to say hi, see:
https://heyguys.cc/

Oh, copy that. Sorry for my poor english.
msg361987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-14 07:48
New changeset d212c3c55d414203b0579e000d9f340f8cd11be7 by Dong-hee Na in branch 'master':
bpo-39573: PyXXX_Check() macros use Py_IS_TYPE() (GH-18508)
https://github.com/python/cpython/commit/d212c3c55d414203b0579e000d9f340f8cd11be7
msg361988 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-14 07:50
New changeset 9aeb0ef9309384099e2f23bcee2240fbc096568e by Dong-hee Na in branch 'master':
 bpo-39573: Update clinic to use Py_IS_TYPE() function (GH-18507)
https://github.com/python/cpython/commit/9aeb0ef9309384099e2f23bcee2240fbc096568e
msg362033 - (view) Author: Andy Lester (petdance) * Date: 2020-02-15 21:34
@vstinner would it be helpful if I went on a sweep looking for places we can use the new Py_IS_TYPE macro?

Getting away from Py_TYPE(op) would also mean a move to making the internals const-correct.
msg362034 - (view) Author: Andy Lester (petdance) * Date: 2020-02-15 21:36
I'm hoping that a goal here is to make

static inline int _Py_IS_TYPE(PyObject *ob, PyTypeObject *type)

actually be

static inline int _Py_IS_TYPE(const PyObject *ob, const PyTypeObject *type)
msg362133 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-17 09:48
> Getting away from Py_TYPE(op) would also mean a move to making the internals const-correct.

Would you mind to explain how it's an issue to modify PyObject* temporarily during a function call? It's common to increase the object reference count to ensure that it doesn't go even while we use it.
msg362134 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-17 10:09
New changeset 1b55b65638254aa78b005fbf0b71fb02499f1852 by Dong-hee Na in branch 'master':
bpo-39573: Clean up modules and headers to use Py_IS_TYPE() function (GH-18521)
https://github.com/python/cpython/commit/1b55b65638254aa78b005fbf0b71fb02499f1852
msg362166 - (view) Author: Andy Lester (petdance) * Date: 2020-02-18 02:21
> Would you mind to explain how it's an issue to modify PyObject* temporarily during a function call?

It's not a problem to modify the PyObject* during a function call.  However, many functions don't need to modify the object, but are still taking non-const PyObject* arguments.

For example if I have this code:

    if (Py_TYPE(deque) == &deque_type) {

That doesn't modify deque to check the type, but because Py_TYPE casts away the constness, deque can't be a const object.

However, with the new Py_IS_TYPE function:

    if (Py_IS_TYPE(deque, &deque_type)) {

and these two changes:

-static inline int _Py_IS_TYPE(PyObject *ob, PyTypeObject *type) {
+static inline int _Py_IS_TYPE(const PyObject *ob, const PyTypeObject *type) {
     return ob->ob_type == type;
 }
-#define Py_IS_TYPE(ob, type) _Py_IS_TYPE(_PyObject_CAST(ob), type)
+#define Py_IS_TYPE(ob, type) _Py_IS_TYPE(((const PyObject*)(ob)), type)

the deque variable can be const.

Another example of a common pattern that I believe could benefit from this is Py_TYPE(ob)->tp_name.  That could be turned into Py_TYPE_NAME(ob) and that would allow the ob to be a const pointer. 

If we can keep functions that don't modify the object to accept const PyObject* it will help make things safer in the long run.
msg362212 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-02-18 13:50
> If we can keep functions that don't modify the object to accept const PyObject* it will help make things safer in the long run.

In my experience, trying to add "const" is quite painful, since the "const" has to be propagated to all functions called by the modified function. Python never used "const" with "PyObject*" because they are *so many* functions which really modify objects on purpose.

I don't see how adding "const" would help this issue "Make PyObject an opaque structure in the limited C API". If you consider that something should be changed, please open a *separated* issue.
msg362216 - (view) Author: Andy Lester (petdance) * Date: 2020-02-18 14:38
All I'm saying is that I think Py_IS_TYPE is a great idea, and that Py_IS_TYPE should take const arguments, since its arguments are not modified.  If you think that should go in a different ticket, then I can make that happen.
msg362445 - (view) Author: Andy Lester (petdance) * Date: 2020-02-22 04:45
Just added a new PR to finish off the remaining places to use Py_IS_TYPE()

https://github.com/python/cpython/pull/18601
msg363345 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-04 13:15
New changeset dffe4c07095e0c693e094d3c140e85a68bd8128e by Andy Lester in branch 'master':
bpo-39573: Finish converting to new Py_IS_TYPE() macro (GH-18601)
https://github.com/python/cpython/commit/dffe4c07095e0c693e094d3c140e85a68bd8128e
msg363494 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-06 08:04
New changeset 8767ce92d24d3687405848442e6c67cf0af1c657 by Andy Lester in branch 'master':
bpo-39573: Make Py_IS_TYPE() take constant parameters (GH-18799)
https://github.com/python/cpython/commit/8767ce92d24d3687405848442e6c67cf0af1c657
msg363564 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-06 22:53
New changeset 557287075c264d2458cd3e1b45e9b8ee5341e0a1 by Andy Lester in branch 'master':
 bpo-39573: Use Py_IS_TYPE() macro to check for types (GH-18809)
https://github.com/python/cpython/commit/557287075c264d2458cd3e1b45e9b8ee5341e0a1
msg365690 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-03 11:57
I created bpo-40170 "[C API] Make PyTypeObject structure an opaque structure in the public C API".
msg366473 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-15 01:35
PyType_FromSpec() and PyType_Spec API are not currently compatible with opaque PyObject.

Example:
---
#define PyObject_HEAD    PyObject ob_base;

typedef struct {
    PyObject_HEAD
    ...
} MyObject;

static PyType_Spec type_spec = {
    .name = "MyObject",
    .basicsize = sizeof(MyObject),
    ...
};

... = PyType_FromSpec(&type_spec);
---

sizeof(MyObject) requires to compute sizeof(PyObject).

Issue reported by Ronald Oussoren on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/message/PGKRW7S2IUOWVRX6F7RT6VAWD3ZPUDYS/
msg366493 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-04-15 08:30
The incompatibility mentioned in msg366473 is probably fixable by treating the PyObject header the same as the GC head structure. With some care this could mostly maintain binary compatibility by inserting some unused fields in PyObject_HEAD instead of the PyObject header when an extension targets a stable ABI version that has the PyObject header in-line.

This issue seems to be comparible to the "fragile instance variable" issue fixed in Objective-C 2.0, see <https://en.wikipedia.org/wiki/Objective-C#Non-fragile_instance_variables>.  That was fixed by adding a level of indirection when accessing member variables.

Something like that is probably necessary to be able to subclass builtin types (other than object itself) in an extension.
msg368047 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-05-04 13:32
New changeset 5e8ffe147710e449c2e935a4e2ff5cbd19828a8a by Hai Shi in branch 'master':
bpo-39573: Use Py_IS_TYPE to check for types (GH-19882)
https://github.com/python/cpython/commit/5e8ffe147710e449c2e935a4e2ff5cbd19828a8a
msg369896 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-05-25 16:52
New changeset ad3252bad905d41635bcbb4b76db30d570cf0087 by Dong-hee Na in branch 'master':
bpo-39573: Convert Py_TYPE() to a static inline function (GH-20290)
https://github.com/python/cpython/commit/ad3252bad905d41635bcbb4b76db30d570cf0087
msg369898 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2020-05-25 17:25
New changeset 7d847e29d76b178c2db66b180065771b4d90c78f by Dong-hee Na in branch 'master':
bpo-39573: Fix buildbot failure for tupleobject.c (GH-20391)
https://github.com/python/cpython/commit/7d847e29d76b178c2db66b180065771b4d90c78f
msg370074 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-27 12:55
New changeset fe2978b3b940fe2478335e3a2ca5ad22338cdf9c by Victor Stinner in branch 'master':
bpo-39573: Convert Py_REFCNT and Py_SIZE to functions (GH-20429)
https://github.com/python/cpython/commit/fe2978b3b940fe2478335e3a2ca5ad22338cdf9c
msg370303 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-29 12:24
> bpo-39573: Convert Py_TYPE() to a static inline function (GH-20290)

This change broke two projects:

* Cython: https://github.com/cython/cython/commit/d8e93b332fe7d15459433ea74cd29178c03186bd (FIXED)
* immutables: https://github.com/MagicStack/immutables/issues/46
msg370638 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-03 00:18
> bpo-39573: Convert Py_TYPE() to a static inline function (GH-20290)

This change broke pycurl:
https://github.com/pycurl/pycurl/pull/636

Extract of its current code:
"""
    /* Initialize the type of the new type objects here; doing it here
     * is required for portability to Windows without requiring C++. */
    p_Curl_Type = &Curl_Type;
    p_CurlMulti_Type = &CurlMulti_Type;
    p_CurlShare_Type = &CurlShare_Type;
    Py_TYPE(&Curl_Type) = &PyType_Type; 
    Py_TYPE(&CurlMulti_Type) = &PyType_Type; 
    Py_TYPE(&CurlShare_Type) = &PyType_Type; 
"""
msg370663 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-03 12:43
To port code to Python 3.10, the following macro can be copied/pasted in your code. It defines Py_SET_SIZE() if it's not defined.

#if PY_VERSION_HEX < 0x030900A4
#  define Py_SET_SIZE(obj, size) do { Py_SIZE(obj) = (size); } while (0)
#endif
msg370665 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-03 12:44
Similar macro for Py_SET_TYPE:

#if PY_VERSION_HEX < 0x030900A4
#  define Py_SET_TYPE(obj, size) do { Py_TYPE(obj) = (size); } while (0)
#endif
msg370666 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-03 12:46
numpy fix defines Py_SET_TYPE() and Py_SET_SIZE() on old Python versions:

* https://github.com/numpy/numpy/commit/a96b18e3d4d11be31a321999cda4b795ea9eccaa
* https://github.com/numpy/numpy/pull/16417

In the whole numpy code base, only 5 lines have to be modified!
msg370671 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-03 13:17
I proposed PR 20610 to enhance the documentation explaining how to port existing to code to Py_SET_SIZE() & cie.
msg370729 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-04 20:10
New changeset dc24b8a2ac32114313bae519db3ccc21fe45c982 by Victor Stinner in branch 'master':
bpo-39573: Porting to Python 3.10: Py_SET_SIZE() macro (GH-20610)
https://github.com/python/cpython/commit/dc24b8a2ac32114313bae519db3ccc21fe45c982
msg370902 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-07 14:24
Note: numpy was updated to also the use the macros using ", (void)0":
https://github.com/numpy/numpy/commit/f1671076c80bd972421751f2d48186ee9ac808aa
msg370932 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-07 23:41
See also bpo-40881 "--with-valgrind broken": unicode_release_interned() still used "Py_REFCNT(s) += 1;". It's now fixed by commit c96a61e8163c2d25ed4ac77cf96201fd0bdb945c.
msg372308 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 08:54
> Another idea would be to convert some C extensions of the standard library to the limited C API. It would ensure that the limited C API contains enough functions to be useful, but would also notify us directly if the API is broken.

I created bpo-41111: "Convert a few stdlib extensions to the limited C API".
msg373460 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-07-10 10:40
New changeset 8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1 by Victor Stinner in branch 'master':
bpo-39573: Use the Py_TYPE() macro (GH-21433)
https://github.com/python/cpython/commit/8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1
msg379675 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-10-26 17:45
This also breaks pycurl:

https://github.com/pycurl/pycurl/pull/660

And breezy:

https://bugzilla.redhat.com/show_bug.cgi?id=1890880 (not yet reported upstream)


I don't understand the rationale for this change in depth, but does the benefit outweigh (yet another) backwards incompatibility?
msg379679 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2020-10-26 18:27
> I don't understand the rationale for this change in depth, but
> does the benefit outweigh (yet another) backwards incompatibility?

I think you can have it both ways.  Do you want a C API that is
stable over a long period of CPython releases or do you want to
continue to be able to look deep (i.e. non opaque PyObject*) into
CPython implementation internals?

During the sprint last week, we talked about how to provide a
compatible API, similar to what Pypy cpyext does.  It would be
possible to provide a (nearly) fully compatible API with the
approach.  It could get quite painful for CPython to maintain such a
thing however.  E.g. cpyext has proxy objects (to maintain CPython
compatible structure layouts) but keeping those proxies in sync with
the internal VM object structures is expensive and tricky.

Certainly making PyObject opaque is going to break some 3rd party
code.  Making it opaque for the non-limited API is not an option
IMHO because it breaks too much 3rd party code.   Is making it
opaque for the limited C API going to break too much code?  Maybe, I
don't know.  Thanks for pointing out pycurl and breezy.
msg379680 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2020-10-26 18:28
Correction: I think you *cannot* have it both ways.
msg379757 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-27 13:55
Miro:
> This also breaks pycurl:
> https://github.com/pycurl/pycurl/pull/660

Right, see my previous comment, another PR was already proposed in May!

me:
> This change broke pycurl:
> https://github.com/pycurl/pycurl/pull/636

Merged pycurl fix:
https://github.com/pycurl/pycurl/commit/e633f9a1ac4df5e249e78c218d5fbbd848219042
msg379759 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-10-27 14:11
Miro:
> I don't understand the rationale for this change in depth, but does the benefit outweigh (yet another) backwards incompatibility?

I wrote PEP 620 "Hide implementation details from the C API" to explain the rationale:
https://www.python.org/dev/peps/pep-0620/
msg381337 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-11-18 13:31
Another batch of broken projects:

PyPAM https://bugzilla.redhat.com/show_bug.cgi?id=1897264
bitarray https://bugzilla.redhat.com/show_bug.cgi?id=1897536
boost https://bugzilla.redhat.com/show_bug.cgi?id=1896382
duplicity https://bugzilla.redhat.com/show_bug.cgi?id=1896684
gobject-introspection https://bugzilla.redhat.com/show_bug.cgi?id=1893194
mercurial https://bugzilla.redhat.com/show_bug.cgi?id=1897178
pybluez https://bugzilla.redhat.com/show_bug.cgi?id=1897256
pygobject3 https://bugzilla.redhat.com/show_bug.cgi?id=1894522
pylibacl https://bugzilla.redhat.com/show_bug.cgi?id=1897529
pyside2 https://bugzilla.redhat.com/show_bug.cgi?id=1898974
rdiff-backup https://bugzilla.redhat.com/show_bug.cgi?id=1898980



Those are just the initial set of packages we have discovered so far. I think there will be more.
msg381345 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-18 14:24
I wrote PR 23366 to revert the Py_TYPE() and Py_SIZE() changes: convert them back to macros to allow again "Py_TYPE(obj) = type;" and "Py_SIZE(obj) = size;" syntaxes.


Miro Hrončok:
> Another batch of broken projects:
> PyPAM https://bugzilla.redhat.com/show_bug.cgi?id=1897264

"Py_TYPE="

> bitarray https://bugzilla.redhat.com/show_bug.cgi?id=1897536

"Py_TYPE=/Py_SIZE="

> boost https://bugzilla.redhat.com/show_bug.cgi?id=1896382

"Py_TYPE="

> duplicity https://bugzilla.redhat.com/show_bug.cgi?id=1896684

"Py_TYPE="

> gobject-introspection https://bugzilla.redhat.com/show_bug.cgi?id=1893194

"Py_TYPE="

> mercurial https://bugzilla.redhat.com/show_bug.cgi?id=1897178

"Py_TYPE=/Py_SIZE="

> pybluez https://bugzilla.redhat.com/show_bug.cgi?id=1897256

"Py_TYPE="

> pygobject3 https://bugzilla.redhat.com/show_bug.cgi?id=1894522

"Py_TYPE="

> pylibacl https://bugzilla.redhat.com/show_bug.cgi?id=1897529

"Py_TYPE="

> pyside2 https://bugzilla.redhat.com/show_bug.cgi?id=1898974

"Py_TYPE="

> rdiff-backup https://bugzilla.redhat.com/show_bug.cgi?id=1898980

"Py_TYPE="



> Those are just the initial set of packages we have discovered so far. I think there will be more.

Well, since the PEP 620 is not accepted, I prefer to revert the Py_TYPE() and the Py_SIZE() changes for now. We should have a wider discussion on how to introduce incompatible changes into the C API before being able to push more incompatible changes which impact a so wide range of C extension modules.

One practical problem is how to estimate the number of broken Python projects to decide if an incompatible change can be introduced or not. When I did early experiment before merging the PR 20290, it seems like only a minority of C extensions rely on "Py_TYPE(obj) = type;" syntax. It's a common pattern to define a type statically. Pseudo-code:

---
PyTypeObject MyType = {...};

PyInit_MyExtension(...)
{
   Py_TYPE(&MyType) = ...;
   PyType_Ready(&MyType);
   ...
}
---

"Py_TYPE(&MyType) = ...;" is required since some C compilers don't support setting ob_type directly in the MyType static declaration. The type must be set at runtime.

Also I considered that the change is trivial enough to be accepable. Well, I was wrong, and that's why I'm not proposing to revert thes changes.

About the rationale for introducing C API incompatible changes, see the PEP 620.
msg381365 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-18 17:48
New changeset 0e2ac21dd4960574e89561243763eabba685296a by Victor Stinner in branch 'master':
bpo-39573: Convert Py_TYPE() and Py_SIZE() back to macros (GH-23366)
https://github.com/python/cpython/commit/0e2ac21dd4960574e89561243763eabba685296a
msg381374 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-18 21:39
New changeset e0251787d85950538cf2490c2c73cc680b153940 by Miro Hrončok in branch 'master':
bpo-39573: Remove What's new entry for Py_SIZE() (GH-23375)
https://github.com/python/cpython/commit/e0251787d85950538cf2490c2c73cc680b153940
msg381403 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-19 10:58
> And breezy:
> https://bugzilla.redhat.com/show_bug.cgi?id=1890880 (not yet reported upstream)

Oh, I didn't notice that this project is broken by the Py_REFCNT() change. I expected it to be broken by the Py_TYPE() change as others.

Should we revert the Py_REFCNT() change as well? So far, breezy is the only impacted project, and the fix should be simple, no?


breezy uses "Py_REFCNT(self) -= 1;" instead of "Py_DECREF(self);" in its StaticTuple_Intern() function:

static StaticTuple *
StaticTuple_Intern(StaticTuple *self)
{
    PyObject *canonical_tuple = NULL;

    if (_interned_tuples == NULL || _StaticTuple_is_interned(self)) {
        Py_INCREF(self);
        return self;
    }
    /* SimpleSet_Add returns whatever object is present at self
     * or the new object if it needs to add it.
     */
    canonical_tuple = SimpleSet_Add(_interned_tuples, (PyObject *)self);
    if (!canonical_tuple) {
        // Some sort of exception, propogate it.
        return NULL;
    }
    if (canonical_tuple != (PyObject *)self) {
        // There was already a tuple with that value
        return (StaticTuple *)canonical_tuple;
    }
    self->flags |= STATIC_TUPLE_INTERNED_FLAG;
    // The two references in the dict do not count, so that the StaticTuple
    // object does not become immortal just because it was interned.
    Py_REFCNT(self) -= 1;
    return self;
}

But it also uses "Py_REFCNT(self) = 2;" to "revive dead object temporarily for Discard".

static void
StaticTuple_dealloc(StaticTuple *self)
{
    int i, len;

    if (_StaticTuple_is_interned(self)) {
        /* revive dead object temporarily for Discard */
        Py_REFCNT(self) = 2;
        if (SimpleSet_Discard(_interned_tuples, (PyObject*)self) != 1)
            Py_FatalError("deletion of interned StaticTuple failed");
        self->flags &= ~STATIC_TUPLE_INTERNED_FLAG;
    }
    len = self->size;
    for (i = 0; i < len; ++i) {
        Py_XDECREF(self->items[i]);
    }
    Py_TYPE(self)->tp_free((PyObject *)self);
}

It sounds like an optimization using a set of "interned" tuples. Maybe to reduce the memory footprint.
msg381404 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-19 11:06
> And breezy:
> https://bugzilla.redhat.com/show_bug.cgi?id=1890880 (not yet reported upstream)

I reported the issue to breezy upstream:
https://bugs.launchpad.net/brz/+bug/1904868
msg382260 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-01 15:17
I created the https://github.com/pythoncapi/upgrade_pythoncapi project which updates automatically a C extension to newer C API. For example, it uses Py_TYPE(), Py_SIZE(), Py_REFCNT(), Py_SET_SIZE(), Py_SET_SIZE() and Py_SET_REFCNT() functions.
msg382534 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-04 21:51
Status:

* Py_SET_REFCNT(), Py_SET_TYPE() and Py_SET_SIZE() functions added to Python 3.9.
* Python and Cython have been modified to use Py_TYPE(), Py_SET_REFCNT(), Py_IS_TYPE(), etc.
* pythoncapi_compat.h header file has been created to provide new functions to Python 3.6: https://github.com/pythoncapi/pythoncapi_compat
* Script has been created to upgrade C extensions to add support for Python 3.10 without losing support for old Python versions: https://github.com/pythoncapi/pythoncapi_compat
* PEP 620 "Hide implementation details from the C API" written
* Py_TYPE() and Py_SIZE() were converted to a static inline function to deny "Py_TYPE(obj) = type;" syntax, but this change has been reverted during Python 3.10 development cycle since it broke too many C extension modules. (msg381337)

TODO:

* Maybe add a new formatter for type names (msg361523)
* Avoid sizeof(PyObject) in PyType_FromSpec() (msg366473)

The purpose of this issue is only to fix the API part. Replacing static inline functions with opaque function calls (stable ABI) is not in the scope of this issue.
msg382539 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-04 22:12
> immutables: https://github.com/MagicStack/immutables/issues/46

I proposed a fix: https://github.com/MagicStack/immutables/pull/52

> mercurial https://bugzilla.redhat.com/show_bug.cgi?id=1897178

I proposed a fix: https://bz.mercurial-scm.org/show_bug.cgi?id=6451
msg382780 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-09 00:30
> pyside2 https://bugzilla.redhat.com/show_bug.cgi?id=1898974

I proposed a fix upstream: https://bugreports.qt.io/browse/PYSIDE-1436 I also wrote a fix in Fedora: https://src.fedoraproject.org/rpms/python-pyside2/pull-request/7 The issue is now tracked in Fedora as: https://bugzilla.redhat.com/show_bug.cgi?id=1902618
msg382781 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-09 00:45
> bitarray https://bugzilla.redhat.com/show_bug.cgi?id=1897536

I created https://github.com/ilanschnell/bitarray/pull/109
msg382783 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-12-09 01:13
> boost https://bugzilla.redhat.com/show_bug.cgi?id=1896382

I proposed https://github.com/boostorg/python/pull/330 fix.

See also https://github.com/boostorg/python/pull/329
msg394954 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-02 23:46
Most projects broken by Py_TYPE and Py_SIZE changes have been fixed since last year. I succeeded to use my new pythoncapi_compat project on multiple C extensions. So I propose again to convert Py_TYPE and Py_SIZE macros to static inline functions: https://github.com/python/cpython/pull/26493

I also proposed to promote the pythoncapi_compat project in the "C API: Porting to Python 3.10" section of "What's New In Python 3.10?" on python-dev:
https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNOYEDUTSPAATUDP55ZSSQM5RRC/

I already announced the pythoncapi_compat project on the capi-sig last December:
https://mail.python.org/archives/list/capi-sig@python.org/thread/LFLXFMKMZ77UCDUFD5EQCONSAFFWJWOZ/
msg394971 - (view) Author: Dong-hee Na (corona10) * (Python committer) Date: 2021-06-03 02:08
> So I propose again to convert Py_TYPE and Py_SIZE macros to static inline functions

+1
msg395018 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-03 16:43
New changeset f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970 by Victor Stinner in branch 'main':
bpo-39573: Py_TYPE becomes a static inline function (GH-26493)
https://github.com/python/cpython/commit/f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970
msg395205 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-06-06 11:12
@victor, git bisect tells me the change f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970 caused test_exceptions.ExceptionTests.test_recursion_in_except_handler to stack overflow only on windows debug builds. 3 windows buildbots using python debug mode is affected. Python compiled with release mode is *not* affected and passes the test. Here's an example error on one of the buildbots:

https://buildbot.python.org/all/#/builders/596/builds/354/steps/4/logs/stdio

I can also reproduce this locally. I tracked this issue down after a recursion in AST also caused a stack overflow, see my message here:
https://bugs.python.org/msg395172

TLDR: Windows builds seems to set stack size to 2MB, on *nix it's probably higher (usually 8MB). I suspect the static inline functions are not being inlined in windows debug builds, so every function call adds to the stack. In that message I proposed to increase the stack size on windows but there are some concerns (see msg395177). What do you think?
msg395206 - (view) Author: William Pickard (WildCard65) * Date: 2021-06-06 11:21
MSVC by default disables method inlining (/Ob0) when '/Od' is specified on the command line while the optimization options specify '/Ob2'.
msg395287 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-07 21:03
Ken Jin: Please open a separated issue for test_exceptions.test_recursion_in_except_handler(). It's not directly related to marking PyObject opaque, as William Pickard explained.

See my notes on the stack size and stack overflow on a recursion error on Windows:
https://pythondev.readthedocs.io/unstable_tests.html#unlimited-recursion
msg395323 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-06-08 11:24
New changeset 6d518bb3a11f9b16098f45b21a13ebe8f537f045 by Pablo Galindo in branch 'main':
bpo-44348: Revert "bpo-39573: Py_TYPE becomes a static inline function (GH-26493)" (GH-26596)
https://github.com/python/cpython/commit/6d518bb3a11f9b16098f45b21a13ebe8f537f045
msg395536 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-06-10 13:27
See also bpo-44378: "Py_IS_TYPE(): cast discards ‘const’ qualifier from pointer target type".

If Py_TYPE() is converted again to a static inline function which takes a "const PyObject*" type, Py_IS_TYPE() can be modified again at the same time to use Py_TYPE().
msg401365 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 09:59
New changeset cb15afcccffc6c42cbfb7456ce8db89cd2f77512 by Victor Stinner in branch 'main':
bpo-39573: Py_TYPE becomes a static inline function (GH-28128)
https://github.com/python/cpython/commit/cb15afcccffc6c42cbfb7456ce8db89cd2f77512
msg401370 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 10:33
> boost https://bugzilla.redhat.com/show_bug.cgi?id=1896382

Fixed by: https://github.com/boostorg/python/commit/500194edb7833d0627ce7a2595fec49d0aae2484
msg401378 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 12:12
I checked again the list of broken projects listed previously.

Fixed:

* Cython: https://github.com/cython/cython/commit/d8e93b332fe7d15459433ea74cd29178c03186bd
* immutables: https://github.com/MagicStack/immutables/pull/52
* numpy:

  * https://github.com/numpy/numpy/commit/a96b18e3d4d11be31a321999cda4b795ea9eccaa
  * https://github.com/numpy/numpy/commit/f1671076c80bd972421751f2d48186ee9ac808aa

* pycurl: https://github.com/pycurl/pycurl/commit/e633f9a1ac4df5e249e78c218d5fbbd848219042
* bitarray: https://github.com/ilanschnell/bitarray/pull/109
* mercurial: https://bz.mercurial-scm.org/show_bug.cgi?id=6451
* boost: https://github.com/boostorg/python/commit/500194edb7833d0627ce7a2595fec49d0aae2484
* pyside2: https://bugreports.qt.io/browse/PYSIDE-1436
* breezy: https://bugs.launchpad.net/brz/+bug/1904868
* duplicity: https://git.launchpad.net/duplicity/commit/duplicity/_librsyncmodule.c?id=bbaae91b5ac6ef7e295968e508522884609fbf84
* gobject-introspection: https://gitlab.gnome.org/GNOME/gobject-introspection/-/merge_requests/243

Fix proposed:

* pybluez: https://github.com/pybluez/pybluez/pull/410

Broken:

* PyPAM
* pygobject3
* pylibacl 
* rdiff-backup
msg401395 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 16:06
At commit cb15afcccffc6c42cbfb7456ce8db89cd2f77512, I am able to rename PyObject members (to make sure that the structure is not accessed directly), I only had to modify header files:

* Py_REFCNT(), Py_SET_REFCNT()
* Py_INCREF(), Py_DECREF()
* Py_TYPE(), Py_SET_TYPE()
* Py_IS_TYPE()

And just two more C files which corner cases:

* 1 line in Python/specialize.c
* 1 line in Modules/_testcapimodule.c: check_pyobject_forbidden_bytes_is_freed()

--

I did the same with PyVarObject, rename the ob_size member. I had to modify header files:

* Py_SIZE(), Py_SET_SIZE()

But I had to modify the following function of the array module:

static int
array_buffer_getbuf(arrayobject *self, Py_buffer *view, int flags)
{
    ...
    if ((flags & PyBUF_ND)==PyBUF_ND) {
        view->shape = &((PyVarObject*)self)->ob_size;
    }
    ...
    return 0;
}

I'm not sure how to patch this function.

--

This experience doesn't check usage of sizeof(PyObject) and sizeof(PyVarObject) which would break if these structures become opaque. sizeof() issues are listed in previous comments.
msg401396 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 16:14
Oh and obviously, it's not possible possible to define structures which *include* PyObject or PyVarObject if PyObject and PyVarObject become opaque. Example:

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

This C code requires the PyObject structure to be fully defined (not being opaque).

A new C API and ABI where structures *don't* include PyObject or PyVarObject should be designed to allocate their members "before" the PyObject* pointer value. Something like the current PyGC_Head structure which is excluded from PyObject and stored *before* the "PyObject*" pointer.

Simplified code which allocates memory for an object implementin the GC protocol:

static PyObject *
_PyObject_GC_Malloc(size_t basicsize)
{
    ...
    size_t size = sizeof(PyGC_Head) + basicsize;
    ...
    PyGC_Head *g = (PyGC_Head *)PyObject_Malloc(size);
    ...
    PyObject *op = (PyObject *)(g + 1);
    return op;
}
msg401399 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 16:32
I changed the issue title to restrict its scope: "[C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type".

Making PyObject and PyVarObject structures opaque is a broader topic which should be splited into sub-issues.

"Py_TYPE(obj)=type;" is now disallowed. I consider that the work of this issue is now completed and I close the issue.

Thanks everyone who help to fix these tedious issues!

You can continue to use this issue if you need my help to adapt your C extensions to Py_SET_TYPE()/Py_SET_SIZE().

See also the upgrade_pythoncapi.py script of the pythoncapi_compat project which helps to port your C extensions without losing support for old Python versions:
https://github.com/pythoncapi/pythoncapi_compat

See also the Py_TYPE() change announcement on the capi-sig list:
https://mail.python.org/archives/list/capi-sig@python.org/thread/WGRLTHTHC32DQTACPPX36TPR2GLJAFRB/
msg403252 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-05 16:36
I wrote an article about these changes:
https://vstinner.github.io/c-api-abstract-pyobject.html

It elaborates the rationale for making these changes.
msg410995 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-01-20 00:24
> @victor, git bisect tells me the change f3fa63ec75fdbb4a08a10957a5c631bf0c4a5970 caused test_exceptions.ExceptionTests.test_recursion_in_except_handler to stack overflow only on windows debug builds.

FYI this regression was handled last year in bpo-44348 "test_exceptions.ExceptionTests.test_recursion_in_except_handler stack overflow on Windows debug builds" and fixed at 2021-09-07 by using the trashcan mecanism in the BaseException deallocator function:

New changeset fb305092a5d7894b41f122c1a1117b3abf4c567e by Victor Stinner in branch 'main':
bpo-44348: BaseException deallocator uses trashcan (GH-28190)
https://github.com/python/cpython/commit/fb305092a5d7894b41f122c1a1117b3abf4c567e
History
Date User Action Args
2022-04-11 14:59:26adminsetgithub: 83754
2022-01-20 00:24:00vstinnersetnosy: + vstinner
messages: + msg410995
2021-11-04 13:58:23eryksunsetnosy: - ahmedsayeed1982
-> (no value)
2021-11-04 13:58:10eryksunsetmessages: - msg405687
2021-11-04 12:08:29ahmedsayeed1982setnosy: + ahmedsayeed1982, - nascheme, ronaldoussoren, vstinner, serhiy.storchaka, steve.dower, hroncok, corona10, ZackerySpytz, pablogsal, WildCard65, BTaskaya, shihai1991, erlendaasland, kj
messages: + msg405687
2021-10-05 16:36:53vstinnersetmessages: + msg403252
2021-09-08 16:32:15vstinnersetstatus: open -> closed
versions: + Python 3.11, - Python 3.9
title: [C API] Make PyObject an opaque structure in the limited C API -> [C API] Avoid accessing PyObject and PyVarObject members directly: add Py_SET_TYPE() and Py_IS_TYPE(), disallow Py_TYPE(obj)=type
messages: + msg401399

resolution: fixed
stage: patch review -> resolved
2021-09-08 16:14:01vstinnersetmessages: + msg401396
2021-09-08 16:06:14vstinnersetmessages: + msg401395
2021-09-08 12:12:35vstinnersetmessages: + msg401378
2021-09-08 10:33:11vstinnersetmessages: + msg401370
2021-09-08 09:59:21vstinnersetmessages: + msg401365
2021-09-02 15:46:06vstinnersetpull_requests: + pull_request26567
2021-06-10 13:27:28vstinnersetmessages: + msg395536
2021-06-08 11:24:57pablogsalsetmessages: + msg395323
2021-06-08 11:07:12pablogsalsetnosy: + pablogsal
pull_requests: + pull_request25180
2021-06-07 21:24:48erlendaaslandsetnosy: + erlendaasland
2021-06-07 21:03:04vstinnersetmessages: + msg395287
2021-06-06 11:21:15WildCard65setmessages: + msg395206
2021-06-06 11:12:42kjsetnosy: + kj
messages: + msg395205
2021-06-06 10:59:47BTaskayasetnosy: + BTaskaya
pull_requests: + pull_request25148
2021-06-03 16:43:09vstinnersetmessages: + msg395018
2021-06-03 02:08:51corona10setmessages: + msg394971
2021-06-02 23:46:36vstinnersetmessages: + msg394954
2021-06-02 23:08:20vstinnersetpull_requests: + pull_request25089
2020-12-09 01:13:17vstinnersetmessages: + msg382783
2020-12-09 00:45:01vstinnersetmessages: + msg382781
2020-12-09 00:30:44vstinnersetmessages: + msg382780
2020-12-04 22:12:15vstinnersetmessages: + msg382539
2020-12-04 21:51:45vstinnersetmessages: + msg382534
2020-12-01 15:36:26petdancesetnosy: - petdance
2020-12-01 15:17:44vstinnersetmessages: + msg382260
2020-11-19 11:06:21vstinnersetmessages: + msg381404
2020-11-19 10:58:56vstinnersetmessages: + msg381403
2020-11-18 21:39:09vstinnersetmessages: + msg381374
2020-11-18 18:49:38hroncoksetpull_requests: + pull_request22268
2020-11-18 17:48:16vstinnersetmessages: + msg381365
2020-11-18 14:24:55vstinnersetmessages: + msg381345
2020-11-18 14:02:52vstinnersetpull_requests: + pull_request22259
2020-11-18 13:31:16hroncoksetmessages: + msg381337
2020-10-27 14:11:28vstinnersetmessages: + msg379759
2020-10-27 13:55:36vstinnersetmessages: + msg379757
2020-10-26 18:28:26naschemesetmessages: + msg379680
2020-10-26 18:27:40naschemesetmessages: + msg379679
2020-10-26 17:45:02hroncoksetnosy: + hroncok
messages: + msg379675
2020-07-10 10:40:54vstinnersetmessages: + msg373460
2020-07-10 10:17:32vstinnersetpull_requests: + pull_request20580
2020-07-01 16:04:14WildCard65setnosy: + WildCard65
pull_requests: + pull_request20410
2020-06-25 08:54:37vstinnersetmessages: + msg372308
2020-06-07 23:41:23vstinnersetmessages: + msg370932
2020-06-07 14:24:18vstinnersetmessages: + msg370902
2020-06-04 20:10:47vstinnersetmessages: + msg370729
2020-06-03 13:17:01vstinnersetmessages: + msg370671
2020-06-03 13:00:52vstinnersetpull_requests: + pull_request19838
2020-06-03 12:46:39vstinnersetmessages: + msg370666
2020-06-03 12:44:23vstinnersetmessages: + msg370665
2020-06-03 12:43:04vstinnersetmessages: + msg370663
2020-06-03 00:18:04vstinnersetmessages: + msg370638
2020-05-29 12:24:24vstinnersetmessages: + msg370303
2020-05-27 12:55:16vstinnersetmessages: + msg370074
2020-05-26 14:19:53vstinnersetpull_requests: + pull_request19686
2020-05-25 17:25:35corona10setmessages: + msg369898
2020-05-25 17:00:03corona10setpull_requests: + pull_request19654
2020-05-25 16:52:57corona10setmessages: + msg369896
2020-05-21 15:37:24corona10setpull_requests: + pull_request19565
2020-05-07 10:21:10ZackerySpytzsetnosy: + ZackerySpytz
pull_requests: + pull_request19294
2020-05-04 13:32:01corona10setmessages: + msg368047
2020-05-03 12:40:35shihai1991setpull_requests: + pull_request19194
2020-04-15 08:30:11ronaldoussorensetnosy: + ronaldoussoren
messages: + msg366493
2020-04-15 01:35:27vstinnersetmessages: + msg366473
title: Make PyObject an opaque structure in the limited C API -> [C API] Make PyObject an opaque structure in the limited C API
2020-04-03 11:57:17vstinnersetmessages: + msg365690
2020-03-06 22:53:21vstinnersetmessages: + msg363564
2020-03-06 16:51:37petdancesetpull_requests: + pull_request18167
2020-03-06 09:43:06vstinnersetpull_requests: + pull_request18161
2020-03-06 08:04:04vstinnersetmessages: + msg363494
2020-03-06 05:52:26petdancesetpull_requests: + pull_request18155
2020-03-06 05:51:27petdancesetpull_requests: + pull_request18154
2020-03-05 05:28:43petdancesetpull_requests: + pull_request18146
2020-03-04 13:15:30vstinnersetmessages: + msg363345
2020-02-22 04:45:50petdancesetmessages: + msg362445
2020-02-22 04:44:36petdancesetpull_requests: + pull_request17968
2020-02-18 14:38:54petdancesetmessages: + msg362216
2020-02-18 13:50:56vstinnersetmessages: + msg362212
2020-02-18 02:21:19petdancesetmessages: + msg362166
2020-02-17 10:09:24vstinnersetmessages: + msg362134
2020-02-17 09:48:51vstinnersetmessages: + msg362133
2020-02-16 17:25:05corona10setpull_requests: + pull_request17898
2020-02-15 21:36:05petdancesetmessages: + msg362034
2020-02-15 21:34:35petdancesetnosy: + petdance
messages: + msg362033
2020-02-14 07:50:23vstinnersetmessages: + msg361988
2020-02-14 07:48:34vstinnersetmessages: + msg361987
2020-02-14 01:21:52corona10setpull_requests: + pull_request17884
2020-02-14 01:11:53corona10setpull_requests: + pull_request17883
2020-02-14 00:50:55shihai1991setmessages: + msg361977
2020-02-13 21:50:34vstinnersetmessages: + msg361971
2020-02-13 17:37:20vstinnersetmessages: + msg361965
2020-02-13 17:35:08vstinnersetmessages: + msg361964
2020-02-13 16:03:41shihai1991setmessages: + msg361963
2020-02-13 15:58:01vstinnersetmessages: + msg361961
2020-02-13 15:55:19shihai1991setnosy: + shihai1991
messages: + msg361960
2020-02-13 01:43:16brandtbuchersetpull_requests: + pull_request17870
2020-02-12 16:54:09corona10setpull_requests: + pull_request17861
2020-02-12 15:36:10corona10setnosy: + corona10
messages: + msg361904
2020-02-09 05:48:39shihai1991setpull_requests: + pull_request17795
2020-02-08 23:46:05vstinnersetmessages: + msg361639
2020-02-08 14:28:50vstinnersetmessages: + msg361631
2020-02-08 12:08:44steve.dowersetnosy: + steve.dower
messages: + msg361626
2020-02-08 03:53:01corona10setpull_requests: + pull_request17785
2020-02-07 22:18:30vstinnersetmessages: + msg361611
2020-02-07 18:45:00vstinnersetmessages: + msg361607
2020-02-07 11:31:10serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg361593
2020-02-07 11:15:57vstinnersetpull_requests: + pull_request17778
2020-02-07 11:05:16vstinnersetmessages: + msg361590
2020-02-07 10:20:39vstinnersetpull_requests: + pull_request17776
2020-02-07 10:18:42vstinnersetmessages: + msg361557
2020-02-07 09:55:12vstinnersetmessages: + msg361555
2020-02-07 09:47:42vstinnersetpull_requests: + pull_request17774
2020-02-07 09:19:24vstinnersetmessages: + msg361549
2020-02-07 08:17:11vstinnersetmessages: + msg361540
2020-02-07 02:39:37vstinnersetpull_requests: + pull_request17770
2020-02-07 02:37:11vstinnersetmessages: + msg361531
2020-02-07 02:06:14vstinnersetpull_requests: + pull_request17769
2020-02-07 02:04:30vstinnersetmessages: + msg361529
2020-02-07 01:42:16vstinnersetmessages: + msg361527
2020-02-07 01:26:55vstinnersetpull_requests: + pull_request17767
2020-02-07 01:24:55vstinnersetmessages: + msg361526
2020-02-07 01:02:14vstinnersetmessages: + msg361523
2020-02-07 00:55:54vstinnersetpull_requests: + pull_request17765
2020-02-07 00:53:27vstinnersetmessages: + msg361522
2020-02-07 00:26:52vstinnersetpull_requests: + pull_request17764
2020-02-07 00:24:52vstinnersetnosy: + nascheme
2020-02-07 00:24:36vstinnersetmessages: + msg361519
2020-02-07 00:22:52vstinnersetmessages: + msg361518
2020-02-07 00:19:37vstinnersetmessages: + msg361517
2020-02-07 00:11:48vstinnersetmessages: + msg361516
2020-02-06 23:41:28vstinnersetpull_requests: + pull_request17763
2020-02-06 23:39:12vstinnersetmessages: + msg361515
2020-02-06 23:38:40vstinnersetmessages: + msg361514
2020-02-06 23:16:05vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request17762
2020-02-06 23:07:12vstinnercreate