This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: [C API] Add private "CAST" macros to clean up casts in C code
Type: Stage: patch review
Components: C API Versions: Python 3.11
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, corona10, gregory.p.smith, kj, vstinner
Priority: normal Keywords: patch

Created on 2022-03-30 12:28 by vstinner, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 32190 merged vstinner, 2022-03-30 12:49
PR 32191 merged vstinner, 2022-03-30 13:25
PR 32192 merged vstinner, 2022-03-30 13:42
PR 32210 open vstinner, 2022-03-31 08:14
Messages (14)
msg416350 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 12:28
Last years, I started to add "CAST" macros like:

#define _PyObject_CAST(op) ((PyObject*)(op))
#define _PyType_CAST(op) (assert(PyType_Check(op)), (PyTypeObject*)(op))

Example of usage:

#define PyObject_TypeCheck(ob, type) PyObject_TypeCheck(_PyObject_CAST(ob), type)

These macros avoids parenthesis. Example without CAST macro:

#define PyCFunction_GET_FLAGS(func) \
        (((PyCFunctionObject *)func) -> m_ml -> ml_flags)

Currently, inline cast requires adding parenthesis:

   ((PyCFunctionObject *)func)

IMO it's more readable with a CAST macro:

#define _PyCFunction_CAST(func) ((PyCFunctionObject *)func)
#define PyCFunction_GET_FLAGS(func) \
        (_PyCFunction_CAST(func)->m_ml->ml_flags)


I propose to add more CAST macros.

By the way, the current Python C API is not fully compatible with C++. "(type)expr" C syntax is seen as "old-style cast" in C++ which recommends static_cast<type>(expr), reinterpret_cast<type>(expr), or another kind of cast. But I prefer to discuss C++ in a separated issue ;-) IMO without considering C++, adding CAST macros is worth it for readability.

I am preparing pull requests for add CAST macros and use existing CAST macros.
msg416353 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2022-03-30 13:16
I think that adding macros makes readability worse.

The macro is only more readable if you already know what it does.
If you don't, then you need to look up the macro, and understand the cast in the macro (which is harder than understanding the original cast).


In general, I find the excessive use of macros and tiny inline function obscures the meaning of code, and makes it hard to reason about what the code is doing.

A few well chosen macros (like Py_INCREF(), etc) can definitely help readability, but if there are too many then anyone reading the code ends up having to lookup loads of macro definitions to understand the code.



Without the macro, the reader needs to parse the cast, which is admittedly a
msg416354 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2022-03-30 13:20
The problem in the example you give is the need for the cast in the first place. If `func` were a `PyCFunctionObject *` instead of a `PyObject *`, then there would be no cast.


Making the casts explicit serves as a reminder that a type check is needed.

```
PyObject *func = ...;
int flags = PyCFunction_GET_FLAGS(func);
```

is dangerous.

```
PyObject *obj = ...;
if (PyCFunction_Check(obj)) {
    PyCFunctionObject *func = (PyCFunctionObject *)obj;
    int flags = func->m_ml->ml_flags;
```

is safe.
msg416357 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2022-03-30 13:52
> tiny inline function obscures the meaning of code

Sadly I have to agree with Mark for that specific case. I've had to debug a segfault before only because the inline function implicitly cast its arguments, and it was accessing a non-existent member. If it were a macro it would access the struct member directly, and the compiler would be able to catch that and warn me before runtime.

However, for the case of _PyCFunctionObject_CAST, I'm not against it. The assert(PyCFunction_Check) pattern has genuinely helped me catch problems in the case of similar macros.
msg416358 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 14:00
> I think that adding macros makes readability worse.

See the attached PRs, I can remove multiple layers of parenthesis in macros by using CAST macros.

> ```
> PyObject *func = ...;
> int flags = PyCFunction_GET_FLAGS(func);
> ```
>
> is dangerous.

Right.

PEP 670 proposes to remove the cast, but only in the limited C API version 3.11 or newer. Sadly, I don't think that we can remove the cast in the general Python C API, it would simply add too many compiler warnings in existing C extensions which previously built without warnings. (See also PEP 670 discussions on python-dev).

Currently, PyCFunction_GET_FLAGS() doesn't check its argument. The macro documentation is quite explicit about it:

/* Macros for direct access to these values. Type checks are *not*
   done, so use with care. */

My GH-32190 PR adds a check in debug mode. So using PyCFunction_GET_FLAGS() in debug mode is safer with this PR.
msg416361 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 14:15
> By the way, the current Python C API is not fully compatible with C++. (...)

I created bpo-47165 "[C API] Test that the Python C API is compatible with C++".
msg416362 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 14:24
Hum, there are two things and maybe we are not talking about the same thing.

* (A) Modifying macros defined in the Python C API (Include/ header files) doing cast on their arguments to use CAST macros
* (B) Modify C code doing casts to use CAST macros

I created this issue for (A). I propose to do (B), but this part is optional.

Mark, Ken: are you fine with (A)? Is your concern about (B)?

But modifying macros to remove the cast of their argument is out of the scope of this issue. As I wrote, it would add compiler warnings, something that I would like to avoid. See:
https://peps.python.org/pep-0670/#cast-pointer-arguments
msg416363 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 14:33
GH-32192 "Add _PyCFunction_CAST() macro" is special. It's used to define functions in PyTypeObject.tp_methods or PyModuleDef.m_methods.

Casting a function pointer can cause issues with Control Flow Integrity (CFI) implemented in LLVM. The _thread module has an undefined behavior fixed by the commit 9eea6eaf23067880f4af3a130e3f67c9812e2f30 of bpo-33015. Previously, a cast ignored that the callback has no return value, whereas pthread_create() requires a function which as a void* return value.

To fix compiler warnings, the (PyCFunction)func cast was replaced with the (PyCFunction)(void(*)(void))func cast to fix bpo-bpo-33012 GCC 8 compiler warning:

    warning: cast between incompatible function types (...)
msg416365 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2022-03-30 14:37
@Victor,

I'm not against (A) or any of the PRs you proposed. They look fine to me. My concern was with certain static inline functions (there are none here :). The _CAST macros have genuinely helped me debug code before.
msg416372 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-30 15:07
> I've had to debug a segfault before only because the inline function implicitly cast its arguments, and it was accessing a non-existent member. If it were a macro it would access the struct member directly, and the compiler would be able to catch that and warn me before runtime.

This is part of Python C API legacy and as I wrote, it's not going to change soon. The minor enhancement that we can do is to inject an assertion when Python is built in debug mode.
msg416417 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-31 07:27
After reading Mark's comments, I reworked my GH-32190 PR to only use the CAST macros in other macros, not in the C code. The CAST macros are not used in such code pattern:

     else if (PyCFunction_Check(func))
-        return ((PyCFunctionObject*)func)->m_ml->ml_name;
+        return _PyCFunctionObject_CAST(func)->m_ml->ml_name;

In fact, this change doesn't bring any value.
msg416418 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-31 07:59
New changeset c14d7e4b816134b8e93ece4066a86d229631ce96 by Victor Stinner in branch 'main':
bpo-47164: Add _PyASCIIObject_CAST() macro (GH-32191)
https://github.com/python/cpython/commit/c14d7e4b816134b8e93ece4066a86d229631ce96
msg416419 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-31 08:02
New changeset f0bc69485677ae8973685866ada0982976d3878f by Victor Stinner in branch 'main':
bpo-47164: Add _PyCFunction_CAST() macro (GH-32192)
https://github.com/python/cpython/commit/f0bc69485677ae8973685866ada0982976d3878f
msg416420 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2022-03-31 08:03
New changeset 7fc39a21cb85163a456eab91b52e5fe85e7f7e3e by Victor Stinner in branch 'main':
bpo-47164: Add _PyCFunctionObject_CAST() macr (GH-32190)
https://github.com/python/cpython/commit/7fc39a21cb85163a456eab91b52e5fe85e7f7e3e
History
Date User Action Args
2022-04-11 14:59:57adminsetgithub: 91320
2022-04-05 06:59:30gregory.p.smithsetnosy: + gregory.p.smith
2022-04-01 02:51:54corona10setnosy: + corona10
2022-03-31 08:14:18vstinnersetpull_requests: + pull_request30286
2022-03-31 08:03:21vstinnersetmessages: + msg416420
2022-03-31 08:02:46vstinnersetmessages: + msg416419
2022-03-31 07:59:31vstinnersetmessages: + msg416418
2022-03-31 07:27:18vstinnersetmessages: + msg416417
2022-03-30 15:07:13vstinnersetmessages: + msg416372
2022-03-30 14:37:01kjsetmessages: + msg416365
2022-03-30 14:33:08vstinnersetmessages: + msg416363
2022-03-30 14:24:25vstinnersetmessages: + msg416362
2022-03-30 14:15:27vstinnersetmessages: + msg416361
2022-03-30 14:00:23vstinnersetmessages: + msg416358
2022-03-30 13:52:27kjsetnosy: + kj
messages: + msg416357
2022-03-30 13:42:14vstinnersetpull_requests: + pull_request30270
2022-03-30 13:25:01vstinnersetpull_requests: + pull_request30269
2022-03-30 13:20:34Mark.Shannonsetmessages: + msg416354
2022-03-30 13:16:10Mark.Shannonsetnosy: + Mark.Shannon
messages: + msg416353
2022-03-30 12:49:50vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request30268
2022-03-30 12:28:10vstinnercreate