classification
Title: [C API] Convert a few stdlib extensions to the limited C API (PEP 384)
Type: Stage: patch review
Components: C API, Extension Modules Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, fdrake, gustavo, petr.viktorin, pmpp, rhettinger, shihai1991, skip.montanaro, vstinner
Priority: normal Keywords: patch

Created on 2020-06-25 08:54 by vstinner, last changed 2021-04-07 00:05 by skip.montanaro.

Pull Requests
URL Status Linked Edit
PR 25115 closed shihai1991, 2021-03-31 17:24
PR 25151 merged vstinner, 2021-04-02 13:53
PR 25180 merged shihai1991, 2021-04-04 08:31
Messages (24)
msg372307 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 08:54
Python stdlib has around 139 extension modules. Not all of them *need* to use internal C API. I'm interested to try to convert a bunch of them to the limited C API, as part of my work on the PEP 620. IMHO "Eating your own dog food" is a good practice to ensure that the limited C API is usable.

Currently, the stdlib has exactly one extension module using the limited C API: the xxlimited module built with Py_LIMITED_API macro defined as 0x03050000 in setup.py. By the way, maybe Py_LIMITED_API should be defined in xxlimited.c, rather than in setup.py.

xxlimited.c is not built if Python is built in debug mode. I'm not sure why.

The main limitation to use the limited C API for stdlib is Argument Clinic which attempts to always emit the most efficient code, like using the METH_FASTCALL calling convention and use private functions like _PyArg_CheckPositional() or "static _PyArg_Parser _parser".

Argument Clinic could be modified to have an option to only use C API of the limited C API. Cython is working on a similar option (restraint emitted code to the limited C API).

I already tried to convert stdlib extensions to the limited C API in bpo-39573. I found other issues:

* PyTypeObject is opaque and so it's not possible to implement a deallocator function (tp_dealloc) which calls tp_free like: Py_TYPE(self)->tp_free((PyObject*)self);
* _Py_IDENTIFIER() is not part of the limited C API

https://bugs.python.org/issue39573#msg361514
msg372309 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 08:55
See also bpo-27499 "PY_SSIZE_T_CLEAN conflicts with Py_LIMITED_API".
msg372310 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:00
Other missing features of the limited C API:

* bpo-2897: "PyMemberDef missing in limited API / Deprecate structmember.h"
* Stefan Behnel wrote: "Some code cannot even be migrated at all,
e.g. because the entire buffer protocol is missing from it. Some bugs were
only fixed in Py3.9, time will tell if anything else is missing."
  https://mail.python.org/archives/list/python-dev@python.org/message/SENQBEJCJ7NYC72ZZ7BGIEDDBTUOXLI4/
msg372311 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:09
PyType_FromSpec() doesn't support 11 slots:

* tp_dict
* tp_mro
* tp_cache
* tp_subclasses
* tp_weaklist
* tp_vectorcall
* tp_weaklistoffset (see PyMemberDef)
* tp_dictoffset (see PyMemberDef)
* tp_vectorcall_offset (see PyMemberDef)
* bf_getbuffer
* bf_releasebuffer

https://docs.python.org/dev/c-api/type.html#c.PyType_Slot

But it is possible to define tp_weaklistoffset, tp_dictoffset and tp_vectorcall_offset via Py_tp_members slot:
https://docs.python.org/dev/c-api/structures.html#pymemberdef-offsets

* "__dictoffset__" => tp_dictoffset
* "__weaklistoffset__" => tp_weaklistoffset
* "__vectorcalloffset__" => tp_vectorcall_offset

Maybe we can do add new members for the 8 missing slots, especially bf_getbuffer and bf_releasebuffer?

commit f7c4e236429606e1c982cacf24e10fc86ef4462f of bpo-40724 added Py_bf_getbuffer and Py_bf_releasebuffer slots to the C API, but these slots are still not available in the limited C API: see bpo-10181.
msg372315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:15
See also bpo-39123: "PyThread_xxx() not available when using limited API".
msg372316 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:17
See also bpo-23903: "Generate PC/python3.def by scraping headers". Steve Dower suggests to convert macros to functions. I'm not sure which functions are still implemented as macro in the limited C API.
msg372321 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:29
See also bpo-16731: "xxlimited/xxmodule docstrings ambiguous".
msg372324 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:32
See also bpo-41073: "[C API] PyType_GetSlot() should accept static types".
msg372325 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:37
See also ./Tools/scripts/abitype.py script and bpo-10943: "abitype: Need better support to port C extension modules to the stable C API".
msg372327 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:38
See also bpo-40077: "Convert static types to PyType_FromSpec()".
msg372332 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-06-25 09:47
See also bpo-29086: "Document C API that is not part of the limited API".
msg372386 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-06-25 19:22
While this might make for an interesting validation experiment, I don't think the results should be checked in.

Code churn is to be avoided when there is no visible user benefit.  It risks destabilizing code, introducing errors, and making the code less recognizable to the other developers who created and maintained it.
msg376506 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 15:37
Some related changes in modules: posix, _hashopenssl, _ast (Python-ast.c), zlib, _struct, tkinter, _curses_panel.

commit b3966639d28313809774ca3859a347b9007be8d2
Author: Eddie Elizondo <eelizondo@fb.com>
Date:   Tue Nov 5 07:16:14 2019 -0800

    bpo-35381 Remove all static state from posixmodule (GH-15892)
    
    
    
    After #9665, this moves the remaining types in posixmodule to be heap-allocated to make it compatible with PEP384 as well as modifying all the type accessors to fully make the type opaque.
    
    The original PR that got messed up a rebase: https://github.com/python/cpython/pull/10854. All the issues in that commit have now been addressed since https://github.com/python/cpython/pull/11661 got committed.
    
    This change also removes any state from the data segment and onto the module state itself.
    
    
    https://bugs.python.org/issue35381
    
    
    
    Automerge-Triggered-By: @encukou

commit df69e75edcc08475bc9a57a5a76df8a45bfc3c34
Author: Christian Heimes <christian@python.org>
Date:   Wed Sep 25 23:03:30 2019 +0200

    bpo-38142: Updated _hashopenssl.c to be PEP 384 compliant (#16071)
    
    * Updated _hashopenssl.c to be PEP 384 compliant
    * Remove refleak test from test_hashlib. The updated type no longer accepts random arguments to __init__.

commit ac46eb4ad6662cf6d771b20d8963658b2186c48c
Author: Dino Viehland <dinoviehland@fb.com>
Date:   Wed Sep 11 10:16:34 2019 -0700

    bpo-38113: Update the Python-ast.c generator to PEP384 (gh-15957)
    
    Summary: This mostly migrates Python-ast.c to PEP384 and removes all statics from the whole file. This modifies the generator itself that generates the Python-ast.c. It leaves in the usage of _PyObject_LookupAttr even though it's not fully PEP384 compatible (this could always be shimmed in by anyone who needs it).

commit a1ffad07195b8b976f8c371a92028240946d4e76
Author: Dino Viehland <dinoviehland@gmail.com>
Date:   Tue Sep 10 11:27:03 2019 +0100

    bpo-38074: Make zlib extension module PEP-384 compatible (GH-15792)
    
    Updated zlibmodule.c to be PEP 384 compliant.

commit 4f384af067d05b16a554bfd976934fca9f87a1cf
Author: Dino Viehland <dinoviehland@gmail.com>
Date:   Tue Sep 10 11:18:37 2019 +0100

    bpo-38076: Make struct module PEP-384 compatible (#15805)
    
    * PEP-384 _struct
    
    * More PEP-384 fixes for _struct
    
    Summary: Add a couple of more fixes for `_struct` that were previously missed such as removing `tp_*` accessors and using `PyBytesWriter` instead of calling `PyBytes_FromStringAndSize` with `NULL`. Also added a test to confirm that `iter_unpack` type is still uninstantiable.
    
    * 📜🤖 Added by blurb_it.

commit d2217a83d4e2ee9aec1a0bf590820aa77b7ed5e7
Author: Andrew Svetlov <andrew.svetlov@gmail.com>
Date:   Tue Oct 30 22:49:16 2012 +0200

    Issue #15721: apply PEP 384 Refactoring to tkinter module.

commit bc07cb883e5c03b8c108c2c9d86bc0a158d62c27
Author: Martin v. Löwis <martin@v.loewis.de>
Date:   Thu Jun 14 16:01:23 2012 +0200

    Issue #14936: curses_panel was converted to PEP 3121 and PEP 384 API.
    Patch by Robin Schreiber.
msg376507 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 15:40
> See also bpo-40077: "Convert static types to PyType_FromSpec()".

In bpo-1635741, many extension modules have been ported to the multi-phase initilization API (PEP 489), and many static types have been converted to heap types.
msg380415 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2020-11-05 15:31
> Maybe we can do add new members for the 8 missing slots

See: https://bugs.python.org/issue41618?#msg380414
msg389925 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2021-03-31 17:28
> By the way, maybe Py_LIMITED_API should be defined in xxlimited.c, rather than in setup.py.

+1. Defining Py_LIMITED_API in xxlimited.c is more direct than in setup.py. so I created the PR-25115.
msg389959 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-01 07:56
See also bpo-43688: [C API] Support the limited C API in debug mode (Py_INCREF and Py_DECREF).
msg390068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-02 14:48
New changeset 240bcf82a11fe7433a61da70605e924c53b88096 by Victor Stinner in branch 'master':
bpo-41111: xxlimited.c defines Py_LIMITED_API (GH-25151)
https://github.com/python/cpython/commit/240bcf82a11fe7433a61da70605e924c53b88096
msg390159 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2021-04-04 00:16
The latest commit seems to break the build if configured --with-trace-refs.
msg390161 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2021-04-04 00:22
I should revise that comment. The xxlimited and xxlimited_35 modules fail to build. That seems suboptimal, but perhaps is to be expected. Perhaps it would be better that compiling them not be attempted with configuring --with-trace-refs?
msg390179 - (view) Author: hai shi (shihai1991) * (Python triager) Date: 2021-04-04 08:33
> Perhaps it would be better that compiling them not be attempted with configuring --with-trace-refs?

+1. The limited C API can't be build under Py_TRACE_REFS now.
msg390326 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-06 12:55
New changeset 5787ba4a45492e232f5470c7d2e93763198e4b22 by Hai Shi in branch 'master':
bpo-41111: Don't build xxlimited with Py_TRACE_REFS macro (GH-25180)
https://github.com/python/cpython/commit/5787ba4a45492e232f5470c7d2e93763198e4b22
msg390333 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-06 13:55
Skip Montanaro: "The latest commit seems to break the build if configured --with-trace-refs."

Oops, I forgot about this special build mode. Thanks for the reminder Skip, and thanks for the fix Hai ;-)

Skip: By the way, I'm curious, why do you use --with-trace-refs?
msg390389 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2021-04-07 00:05
>
> Skip: By the way, I'm curious, why do you use --with-trace-refs?
>

I'm still horsing around with register opcodes and got in the habit of
building with pydebug and trace refs enabled.
History
Date User Action Args
2021-04-07 00:05:28skip.montanarosetmessages: + msg390389
2021-04-06 13:55:10vstinnersetmessages: + msg390333
2021-04-06 12:55:31vstinnersetmessages: + msg390326
2021-04-04 08:33:46shihai1991setmessages: + msg390179
2021-04-04 08:31:06shihai1991setpull_requests: + pull_request23922
2021-04-04 00:22:24skip.montanarosetmessages: + msg390161
2021-04-04 00:16:06skip.montanarosetnosy: + skip.montanaro
messages: + msg390159
2021-04-02 14:48:20vstinnersetmessages: + msg390068
2021-04-02 13:53:44vstinnersetpull_requests: + pull_request23898
2021-04-01 07:56:33vstinnersetmessages: + msg389959
2021-03-31 17:28:05shihai1991setmessages: + msg389925
2021-03-31 17:24:49shihai1991setkeywords: + patch
stage: patch review
pull_requests: + pull_request23864
2020-11-19 14:24:16fdrakesetnosy: + fdrake
2020-11-19 14:03:11vstinnersetcomponents: + C API
title: Convert a few stdlib extensions to the limited C API (PEP 384) -> [C API] Convert a few stdlib extensions to the limited C API (PEP 384)
2020-11-19 14:03:01vstinnersettitle: Convert a few stdlib extensions to the limited C API -> Convert a few stdlib extensions to the limited C API (PEP 384)
2020-11-05 15:31:52petr.viktorinsetnosy: + petr.viktorin
messages: + msg380415
2020-09-07 15:40:09vstinnersetmessages: + msg376507
2020-09-07 15:37:14vstinnersetmessages: + msg376506
2020-07-05 08:05:03pmppsetnosy: + pmpp
2020-06-28 08:58:17gustavosetnosy: + gustavo
2020-06-26 06:54:02shihai1991setnosy: + shihai1991
2020-06-26 05:38:56SilentGhostsetfiles: - Whatsapp-self-message-delivering-using-python-c55344b7f384e036bbe0fda69918c9eccf146dc1.zip
2020-06-26 01:16:37One Digi Printsetfiles: + Whatsapp-self-message-delivering-using-python-c55344b7f384e036bbe0fda69918c9eccf146dc1.zip
2020-06-26 01:09:00corona10setnosy: + corona10
2020-06-25 19:22:53rhettingersetnosy: + rhettinger
messages: + msg372386
2020-06-25 09:47:41vstinnersetmessages: + msg372332
2020-06-25 09:38:50vstinnersetmessages: + msg372327
2020-06-25 09:37:43vstinnersetmessages: + msg372325
2020-06-25 09:32:50vstinnersetmessages: + msg372324
2020-06-25 09:29:32vstinnersetmessages: + msg372321
2020-06-25 09:17:35vstinnersetmessages: + msg372316
2020-06-25 09:15:40vstinnersetmessages: + msg372315
2020-06-25 09:09:48vstinnersetmessages: + msg372311
2020-06-25 09:00:48vstinnersetmessages: + msg372310
2020-06-25 08:55:21vstinnersetmessages: + msg372309
2020-06-25 08:54:20vstinnercreate