classification
Title: [C API] Hide static types from the limited C API
Type: Stage:
Components: C API, Subinterpreters Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: petr.viktorin, shihai1991, vstinner
Priority: normal Keywords:

Created on 2020-05-11 22:23 by vstinner, last changed 2020-05-26 15:15 by vstinner.

Messages (5)
msg368667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:23
"Statically allocated types" prevents to get per-interpreter GIL: bpo-40512. These types are currently shared by all interpreters.

Eric Snow proposed the idea of creating a heap allocated type in subintepreters. But we should take care of direct usage of the statically allocated type.

For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = {...};". This type is exposed in the limited C API (!) in Include/longobject.c:

PyAPI_DATA(PyTypeObject) PyLong_Type;

It's used but such macro:

#define PyLong_CheckExact(op) Py_IS_TYPE(op, &PyLong_Type)

I don't think that these types are directly accessed in C extensions built with the limited C API. My expectation is that the type is only exposed for "CheckExact" macros.

Currently, 100 statically allocated types are declared in Python header files:

$ grep -F '(PyTypeObject)' Include/ -R
Include/cpython/fileobject.h:PyAPI_DATA(PyTypeObject) PyStdPrinter_Type;
(...)
Include/object.h:PyAPI_DATA(PyTypeObject) PySuper_Type; /* built-in 'super' */
Include/methodobject.h:PyAPI_DATA(PyTypeObject) PyCFunction_Type;

Most of them seem to be exposed in the limited C API.

I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

For "CheckExact" macros, we can continue to provide them in the limited C API but as function calls. So a built C extension would no longer access directly the type, but only do function calls.
msg368668 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:23
See also bpo-40077: "Convert static types to PyType_FromSpec()".
msg368676 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:49
> I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

Hum. How would a C extension subclass the Python int type (PyLong_Type) if it's no longer exposed? One option is to add one function per type, like:

PyObject* Py_GetLongType(void);

It would return a *strong reference* to the type (PyLong_Type).

Another option is to get the type from builtins module or builtins dictionary (PyInterpreterState.builtins). But there is no simple C function to get a builtin object. It requires many calls, handle errors, etc. Maybe a generic helper like the following function would help:

PyObject *Py_GetBuiltin(const char *name);

Note: PyEval_GetBuiltins() exposes the builtins of the *current frame* which maybe not be what you may expect.

Currently, Py_GetBuiltin(name) is not needed since basically *all* Python builtins are *directly* exposed in the C API...
msg368731 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-05-12 14:09
> For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = {...};". This type is exposed in the limited C API (!)

Technically, it is not, see https://www.python.org/dev/peps/pep-0384/#structures
Structures like PyLong_Type are *not* part of the limited API.

> I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

That could only be done in Python 4.0, or if we started C-API 4.0. But I don't think it's necessary here.
msg369999 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 15:15
> Technically, it is not, see https://www.python.org/dev/peps/pep-0384/#structures
> Structures like PyLong_Type are *not* part of the limited API.

The symbol is exported by libpython:

$ objdump -T /lib64/libpython3.8.so.1.0|grep PyLong_Type
000000000030de00 g    DO .data	00000000000001a0  Base        PyLong_Type

A C extension can use a reference to PyLong_Type.

> I don't think it's necessary here.

Did you read my rationale (first message)? Do you mean that per-interpreter GIL is not worth it?

--

A first step would be to expose "CheckExact" macros as function calls in the limited C API.
History
Date User Action Args
2020-05-26 15:15:54vstinnersetmessages: + msg369999
2020-05-15 01:51:54vstinnersetcomponents: + Subinterpreters
2020-05-12 15:52:44shihai1991setnosy: + shihai1991
2020-05-12 14:09:18petr.viktorinsetnosy: + petr.viktorin
messages: + msg368731
2020-05-11 22:49:41vstinnersetmessages: + msg368676
2020-05-11 22:23:47vstinnersetmessages: + msg368668
2020-05-11 22:23:25vstinnercreate