Title: [meta][C API] Avoid C macro pitfalls and usage of static inline functions
Type: Stage: patch review
Components: C API Versions: Python 3.11
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: erlendaasland, lemburg, vstinner
Priority: normal Keywords: patch

Created on 2021-10-15 17:43 by vstinner, last changed 2021-11-23 15:04 by vstinner.

Pull Requests
URL Status Linked Edit
PR 29728 open vstinner, 2021-11-23 15:04
Messages (4)
msg404038 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-10-15 17:43
C macros are really cool and useful, but there are a bunch of pitfalls which are better to avoid:

Some macros of the Python C API have been converted to static inline functions over the last years. It went smoothly, I am not aware of any major issue with these conversions.

This meta issue tracks other issues related to macros and static inline functions.

=== Return void ===

One issue is that some macros are treated as an expression and can be reused, whereas it was not intended. For example PyList_SET_ITEM() was implemented as (simplified code):

  #define PyList_SET_ITEM(op, i, v) (op->ob_item[i] = v)

This expression has a value! Two projects used this value by mistake, like:

  "if (obj == NULL || PyList_SET_ITEM (l, i, obj) < 0)"

PyList_SET_ITEM() was fixed by casting the expression to void:

  #define PyList_SET_ITEM(op, i, v) ((void)(op->ob_item[i] = v))

=> bpo-30459

=== Abuse macros as an l-value ===

The Py_TYPE() macro could be used to assign a value: "Py_TYPE(obj) = new_type".

The macro was defined as:

  #define Py_TYPE(ob) (ob->ob_type)

It was converted to a static inline function to disallow using it as an l-value and a new Py_SET_TYPE(op, new_type) function was added. These changes give more freedom to other Python implementations to implement "PyObject" and Py_SET_TYPE().

=> bpo-45476 "[C API] Disallow using PyFloat_AS_DOUBLE() as l-value"
=> bpo-39573 PyObject Py_TYPE/Py_SET_TYPE

=== C API: Macros and embedded Python ===

Sadly, only symbols exported by libpython are accessible to other programming languages embedding Python. Macros of the Python C API are simply not available to them. Projects embedding Python have to hardcode constants and copy macros to their own language, with the risk of being outdated when Python macros are updated.

Even some projects written in C cannot use macros, because they only use libpython symbols. The vim text editor embeds Python this way.

Also, macros are simply excluded from the Python stable ABI (PEP 384).

=== Performance of static inline functions ===

In bpo-45116, it seems like _PyEval_EvalFrameDefault() reached Visual Studio thresholds and some static inline functions are no longer inlined (Py_INCREF/Py_DECREF).

I also noticed that when Python is built in debug mode in Visual Studio, static inline functions are not inlined. Well, the compiler is free to not inline in debug mode. I guess that GCC and clang also skip inlining using -Og and/or -O0 optimization levels. Using __forceinline and __attribute__((always_inline)) on static inline functions (Py_INCREF, Py_TYPE) for debug builds was discussed in bpo-45094, but the idea was rejected.

On the other side, sometimes it's better to *disable* inlining on purpose to reduce the stack memory consumption, using the Py_NO_INLINE macro. See recent measurements of the stack memory usage:

In GH-28893, I noticed that converting a static inline function (PyObject_CallOneArg) to a regular function made it faster. I am not really sure, more benchmarks should be run to really what's going on.

=== Advantages of static inline functions ===

* It's possible to put a breakpoint on a static inline functions.

* Debuggers and profilers are able to get the static inline function names from the machine line, even with inline functions.

* Parameters and the return value have well defined types.

* Variables have a local scope.

* There is no risk of evaluating an expression multiple times.

* Regular C code. No need to use "\" character to multi-line statement. No need for "do { ... } while (0)" and other quicks to workaround preprocessor pitfalls. No abuse of (((parenthesis))).
msg404045 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2021-10-15 19:18
Meta comment :-) ... wouldn't it be better to enable the Github wiki feature for
such collections ?
msg404183 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-10-18 13:17

See also bpo-43502
msg404185 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-10-18 13:18
Previous discussion on Discourse:
Date User Action Args
2021-11-23 15:04:48vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request27965
2021-10-18 13:18:50erlendaaslandsetmessages: + msg404185
2021-10-18 13:17:23erlendaaslandsetnosy: + erlendaasland
messages: + msg404183
2021-10-15 19:18:44lemburgsetnosy: + lemburg
messages: + msg404045
2021-10-15 17:43:25vstinnercreate