classification
Title: [C API] Hide static types from the limited C API
Type: Stage: patch review
Components: C API, Subinterpreters Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: petr.viktorin, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-05-11 22:23 by vstinner, last changed 2021-02-02 13:12 by petr.viktorin.

Pull Requests
URL Status Linked Edit
PR 24146 open vstinner, 2021-02-01 14:00
Messages (7)
msg368667 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:23
"Statically allocated types" prevents to get per-interpreter GIL: bpo-40512. These types are currently shared by all interpreters.

Eric Snow proposed the idea of creating a heap allocated type in subintepreters. But we should take care of direct usage of the statically allocated type.

For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = {...};". This type is exposed in the limited C API (!) in Include/longobject.c:

PyAPI_DATA(PyTypeObject) PyLong_Type;

It's used but such macro:

#define PyLong_CheckExact(op) Py_IS_TYPE(op, &PyLong_Type)

I don't think that these types are directly accessed in C extensions built with the limited C API. My expectation is that the type is only exposed for "CheckExact" macros.

Currently, 100 statically allocated types are declared in Python header files:

$ grep -F '(PyTypeObject)' Include/ -R
Include/cpython/fileobject.h:PyAPI_DATA(PyTypeObject) PyStdPrinter_Type;
(...)
Include/object.h:PyAPI_DATA(PyTypeObject) PySuper_Type; /* built-in 'super' */
Include/methodobject.h:PyAPI_DATA(PyTypeObject) PyCFunction_Type;

Most of them seem to be exposed in the limited C API.

I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

For "CheckExact" macros, we can continue to provide them in the limited C API but as function calls. So a built C extension would no longer access directly the type, but only do function calls.
msg368668 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:23
See also bpo-40077: "Convert static types to PyType_FromSpec()".
msg368676 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-11 22:49
> I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

Hum. How would a C extension subclass the Python int type (PyLong_Type) if it's no longer exposed? One option is to add one function per type, like:

PyObject* Py_GetLongType(void);

It would return a *strong reference* to the type (PyLong_Type).

Another option is to get the type from builtins module or builtins dictionary (PyInterpreterState.builtins). But there is no simple C function to get a builtin object. It requires many calls, handle errors, etc. Maybe a generic helper like the following function would help:

PyObject *Py_GetBuiltin(const char *name);

Note: PyEval_GetBuiltins() exposes the builtins of the *current frame* which maybe not be what you may expect.

Currently, Py_GetBuiltin(name) is not needed since basically *all* Python builtins are *directly* exposed in the C API...
msg368731 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-05-12 14:09
> For example, Objects/longobject.c defines "PyTypeObject PyLong_Type = {...};". This type is exposed in the limited C API (!)

Technically, it is not, see https://www.python.org/dev/peps/pep-0384/#structures
Structures like PyLong_Type are *not* part of the limited API.

> I propose to break the limited C API backward compatibility on purpose by removing these type definitions form the limited C API.

That could only be done in Python 4.0, or if we started C-API 4.0. But I don't think it's necessary here.
msg369999 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 15:15
> Technically, it is not, see https://www.python.org/dev/peps/pep-0384/#structures
> Structures like PyLong_Type are *not* part of the limited API.

The symbol is exported by libpython:

$ objdump -T /lib64/libpython3.8.so.1.0|grep PyLong_Type
000000000030de00 g    DO .data	00000000000001a0  Base        PyLong_Type

A C extension can use a reference to PyLong_Type.

> I don't think it's necessary here.

Did you read my rationale (first message)? Do you mean that per-interpreter GIL is not worth it?

--

A first step would be to expose "CheckExact" macros as function calls in the limited C API.
msg384539 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-01-06 20:53
PC/python3dll.c exports 66 types in the stable ABI:

Py_GenericAliasType
PyObject_Type
_PyWeakref_CallableProxyType
_PyWeakref_ProxyType
_PyWeakref_RefType
PyBaseObject_Type
PyBool_Type
PyByteArray_Type
PyByteArrayIter_Type
PyBytes_Type
PyBytesIter_Type
PyCallIter_Type
PyCapsule_Type
PyCFunction_Type
PyClassMethodDescr_Type
PyComplex_Type
PyDict_Type
PyDictItems_Type
PyDictIterItem_Type
PyDictIterKey_Type
PyDictIterValue_Type
PyDictKeys_Type
PyDictProxy_Type
PyDictValues_Type
PyEllipsis_Type
PyEnum_Type
PyExc_TypeError
PyFilter_Type
PyFloat_Type
PyFrozenSet_Type
PyGetSetDescr_Type
PyList_Type
PyListIter_Type
PyListRevIter_Type
PyLong_Type
PyLongRangeIter_Type
PyMap_Type
PyMemberDescr_Type
PyMemoryView_Type
PyMethodDescr_Type
PyModule_Type
PyModuleDef_Type
PyNullImporter_Type
PyODict_Type
PyODictItems_Type
PyODictIter_Type
PyODictKeys_Type
PyODictValues_Type
PyProperty_Type
PyRange_Type
PyRangeIter_Type
PyReversed_Type
PySeqIter_Type
PySet_Type
PySetIter_Type
PySlice_Type
PySortWrapper_Type
PySuper_Type
PyTraceBack_Type
PyTuple_Type
PyTupleIter_Type
PyType_Type
PyUnicode_Type
PyUnicodeIter_Type
PyWrapperDescr_Type
PyZip_Type
msg386140 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-02-02 13:12
Sorry, I lost this bug in my TODO list :(

> > I don't think it's necessary here.
> 
> Did you read my rationale (first message)? Do you mean that per-interpreter GIL is not worth it?

Right, I mean that it it is not worth breaking the C-API for all existing modules.
Instead, I think that it can be done as an addition: only modules that don't use things like these static types would be allowed in subinterpreters that have their own GIL.
History
Date User Action Args
2021-02-02 13:12:53petr.viktorinsetmessages: + msg386140
2021-02-01 14:00:51vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23227
2021-01-06 20:53:30vstinnersetmessages: + msg384539
2020-05-26 15:15:54vstinnersetmessages: + msg369999
2020-05-15 01:51:54vstinnersetcomponents: + Subinterpreters
2020-05-12 15:52:44shihai1991setnosy: + shihai1991
2020-05-12 14:09:18petr.viktorinsetnosy: + petr.viktorin
messages: + msg368731
2020-05-11 22:49:41vstinnersetmessages: + msg368676
2020-05-11 22:23:47vstinnersetmessages: + msg368668
2020-05-11 22:23:25vstinnercreate