classification
Title: Convert a few stdlib extensions to the limited C API
Type: Stage:
Components: Extension Modules Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, gustavo, pmpp, rhettinger, shihai1991, vstinner
Priority: normal Keywords:

Created on 2020-06-25 08:54 by vstinner, last changed 2020-07-05 08:05 by pmpp.

Messages (12)
msg372307 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 08:54
Python stdlib has around 139 extension modules. Not all of them *need* to use internal C API. I'm interested to try to convert a bunch of them to the limited C API, as part of my work on the PEP 620. IMHO "Eating your own dog food" is a good practice to ensure that the limited C API is usable.

Currently, the stdlib has exactly one extension module using the limited C API: the xxlimited module built with Py_LIMITED_API macro defined as 0x03050000 in setup.py. By the way, maybe Py_LIMITED_API should be defined in xxlimited.c, rather than in setup.py.

xxlimited.c is not built if Python is built in debug mode. I'm not sure why.

The main limitation to use the limited C API for stdlib is Argument Clinic which attempts to always emit the most efficient code, like using the METH_FASTCALL calling convention and use private functions like _PyArg_CheckPositional() or "static _PyArg_Parser _parser".

Argument Clinic could be modified to have an option to only use C API of the limited C API. Cython is working on a similar option (restraint emitted code to the limited C API).

I already tried to convert stdlib extensions to the limited C API in bpo-39573. I found other issues:

* PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like: Py_TYPE(self)->tp_free((PyObject*)self);
* _Py_IDENTIFIER() is not part of the limited C API

https://bugs.python.org/issue39573#msg361514
msg372309 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 08:55
See also bpo-27499 "PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API".
msg372310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:00
Other missing features of the limited C API:

* bpo-2897: "PyMemberDef missing in limited API / Deprecate structmember.h"
* Stefan Behnel wrote: "Some code cannot even be migrated at all,
e.g. because the entire buffer protocol is missing from it. Some bugs were
only fixed in Py3.9, time will tell if anything else is missing."
  https://mail.python.org/archives/list/python-dev@python.org/message/SENQBEJCJ7NYC72ZZ7BGIEDDBTUOXLI4/
msg372311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:09
PyType_FromSpec() doesn't support 11 slots:

* tp_dict
* tp_mro
* tp_cache
* tp_subclasses
* tp_weaklist
* tp_vectorcall
* tp_weaklistoffset (see PyMemberDef)
* tp_dictoffset (see PyMemberDef)
* tp_vectorcall_offset (see PyMemberDef)
* bf_getbuffer
* bf_releasebuffer

https://docs.python.org/dev/c-api/type.html#c.PyType_Slot

But it is possible to define tp_weaklistoffset, tp_dictoffset and tp_vectorcall_offset via Py_tp_members slot:
https://docs.python.org/dev/c-api/structures.html#pymemberdef-offsets

* "__dictoffset__" => tp_dictoffset
* "__weaklistoffset__" => tp_weaklistoffset
* "__vectorcalloffset__" => tp_vectorcall_offset

Maybe we can do add new members for the 8 missing slots, especially bf_getbuffer and bf_releasebuffer?

commit f7c4e236429606e1c982cacf24e10fc86ef4462f of bpo-40724 added Py_bf_getbuffer and Py_bf_releasebuffer slots to the C API, but these slots are still not available in the limited C API: see bpo-10181.
msg372315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:15
See also bpo-39123: "PyThread_xxx() not available when using limited API".
msg372316 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:17
See also bpo-23903: "Generate PC/python3.def by scraping headers". Steve Dower suggests to convert macros to functions. I'm not sure which functions are still implemented as macro in the limited C API.
msg372321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:29
See also bpo-16731: "xxlimited/xxmodule docstrings ambiguous".
msg372324 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:32
See also bpo-41073: "[C API] PyType_GetSlot() should accept static types".
msg372325 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:37
See also ./Tools/scripts/abitype.py script and bpo-10943: "abitype: Need better support to port C extension modules to the stable C API".
msg372327 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:38
See also bpo-40077: "Convert static types to PyType_FromSpec()".
msg372332 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:47
See also bpo-29086: "Document C API that is not part of the limited API".
msg372386 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-06-25 19:22
While this might make for an interesting validation experiment, I don't think the results should be checked in.

Code churn is to be avoided when there is no visible user benefit.  It risks destabilizing code, introducing errors, and making the code less recognizable to the other developers who created and maintained it.
History
Date User Action Args
2020-07-05 08:05:03pmppsetnosy: + pmpp
2020-06-28 08:58:17gustavosetnosy: + gustavo
2020-06-26 06:54:02shihai1991setnosy: + shihai1991
2020-06-26 05:38:56SilentGhostsetfiles: - Whatsapp-self-message-delivering-using-python-c55344b7f384e036bbe0fda69918c9eccf146dc1.zip
2020-06-26 01:16:37One Digi Printsetfiles: + Whatsapp-self-message-delivering-using-python-c55344b7f384e036bbe0fda69918c9eccf146dc1.zip
2020-06-26 01:09:00corona10setnosy: + corona10
2020-06-25 19:22:53rhettingersetnosy: + rhettinger
messages: + msg372386
2020-06-25 09:47:41vstinnersetmessages: + msg372332
2020-06-25 09:38:50vstinnersetmessages: + msg372327
2020-06-25 09:37:43vstinnersetmessages: + msg372325
2020-06-25 09:32:50vstinnersetmessages: + msg372324
2020-06-25 09:29:32vstinnersetmessages: + msg372321
2020-06-25 09:17:35vstinnersetmessages: + msg372316
2020-06-25 09:15:40vstinnersetmessages: + msg372315
2020-06-25 09:09:48vstinnersetmessages: + msg372311
2020-06-25 09:00:48vstinnersetmessages: + msg372310
2020-06-25 08:55:21vstinnersetmessages: + msg372309
2020-06-25 08:54:20vstinnercreate