classification
Title: Implement PEP 652 -- Maintaining the Stable ABI
Type: Stage: patch review
Components: Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: corona10, erlendaasland, miss-islington, orsenthil, pablogsal, petr.viktorin, vstinner, willingc
Priority: normal Keywords: patch

Created on 2021-04-09 15:43 by petr.viktorin, last changed 2021-06-16 09:53 by miss-islington.

Pull Requests
URL Status Linked Edit
PR 25312 merged petr.viktorin, 2021-04-09 15:47
PR 25315 merged petr.viktorin, 2021-04-09 16:11
PR 25482 merged petr.viktorin, 2021-04-20 13:21
PR 25483 merged petr.viktorin, 2021-04-20 14:30
PR 25668 merged petr.viktorin, 2021-04-27 19:21
PR 25716 closed orsenthil, 2021-04-29 11:59
PR 25920 merged petr.viktorin, 2021-05-05 16:32
PR 25925 merged miss-islington, 2021-05-05 17:32
PR 26034 merged miss-islington, 2021-05-11 14:04
PR 26101 merged petr.viktorin, 2021-05-13 14:59
PR 26123 merged miss-islington, 2021-05-14 09:34
PR 26241 merged petr.viktorin, 2021-05-19 13:31
PR 26353 merged miss-islington, 2021-05-25 11:42
PR 26354 open petr.viktorin, 2021-05-25 12:29
PR 26479 merged petr.viktorin, 2021-06-01 14:06
PR 26603 merged miss-islington, 2021-06-08 15:20
PR 26740 merged petr.viktorin, 2021-06-15 14:52
PR 26753 merged miss-islington, 2021-06-16 09:22
Messages (24)
msg390638 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-09 15:43
This issue tracks implementation of PEP 652.

CPython's Limited C-API and Stable ABI, introduced in PEP 384, will be formalized in a single definitive file, tested, and documented.
msg391081 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-14 14:38
New changeset 92eebf6dd20c541ca5883d010a575fb6ea4a245c by Petr Viktorin in branch 'master':
bpo-43795: Sort PC/python3dll.c (GH-25312)
https://github.com/python/cpython/commit/92eebf6dd20c541ca5883d010a575fb6ea4a245c
msg391217 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-16 15:25
Opened as a separate issue: bpo-43868, Remove PyOS_ReadlineFunctionPointer from the stable ABI list
msg391223 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-16 16:08
I unsure about the following symbols.
Victor, do you think they should be part of the stable ABI?

 - PyThreadState_DeleteCurrent
 - Py_GetArgcArgv
 - PyInterpreterState_GetID
msg391235 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-16 17:23
> PyThreadState_DeleteCurrent

No.

bpo-37878 explicitly excluded it from the limited C API. IMO it's too close to the CPython implementation and so must not be added to the stable ABI.

> Py_GetArgcArgv

No.

Python 3.10 now provides sys.orig_argv which can be read in C.
https://docs.python.org/dev/library/sys.html#sys.orig_argv

Py_GetArgcArgv() is part of the "legacy" C API to initialize Python. The "new" PyConfig API provides PyConfig.orig_argv.

FYI in the PyPI top 4000 projects, the function seems to be called by the following projects:

2021-02-18/uWSGI-2.0.19.1.tar.gz
2021-02-18/stem-1.8.0.tar.gz
2021-02-18/sphinx-gallery-0.8.2.tar.gz
2021-02-18/setproctitle-1.2.2.tar.gz
2021-02-18/pyuwsgi-2.0.19.1.post0.tar.gz
2021-02-18/mercurial-5.7.tar.gz
2021-02-18/httpretty-1.0.5.tar.gz
2021-02-18/CherryPy-18.6.0.tar.gz
2021-02-18/aws-encryption-sdk-cli-2.1.0.tar.gz

> PyInterpreterState_GetID

Yes.

It's part of the limited C API. IMO technically, it's ok to add to the stable ABI. I added it to PC/python3dll.c on purpose. Sadly, it's not available on PyPy, but I don't think that the stable ABI must care about PyPy which is a different use case.
msg391258 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-16 21:02
Thanks! I'll send PRs later.
msg391432 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-20 12:07
Thanks for linking to the issues/documentation!

I see that Py_GetArgcArgv it was exported in python3.def (i.e. stable ABI) in bpo-40910, which got backported to 3.9.

And the removal of PyThreadState_DeleteCurrent was reverted: bpo-38266

While we would like to deprecate/remove them in the future, they are now part of the stable ABI.
They are now in internal headers, so I will exclude them from the limited API.
msg391435 - (view) Author: Erlend E. Aasland (erlendaasland) * (Python triager) Date: 2021-04-20 13:49
Should we mention PEP 652 in Include/README.rst, now that the PEP is accepted?
msg391436 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-20 13:52
> Should we mention PEP 652 in Include/README.rst, now that the PEP is accepted?

No, we should link to the documentation (when it's written). The PEP is a design document; it'll become outdated.
msg391684 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-23 12:14
New changeset e7cc64e297001cc79b9afab80f71d9e6d1267cb7 by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Simplify headers for easier static analysis (GH-25483)
https://github.com/python/cpython/commit/e7cc64e297001cc79b9afab80f71d9e6d1267cb7
msg391686 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-23 12:18
New changeset 9d6a2d0ee7e55402656e1dec46400591b62db331 by Petr Viktorin in branch 'master':
bpo-43795: PEP-652: Clean up the stable ABI/limited API (GH-25482)
https://github.com/python/cpython/commit/9d6a2d0ee7e55402656e1dec46400591b62db331
msg392212 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-28 15:15
Right now, running "make regen-limited-abi" adds again functions which were removed from Doc/data/stable_abi.dat:
https://github.com/python/cpython/pull/25687#issuecomment-828520575

diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
index cdc7160250..6fe61743d6 100644
--- a/Doc/data/stable_abi.dat
+++ b/Doc/data/stable_abi.dat
@@ -351,11 +351,17 @@ PyMapping_Length
 PyMapping_SetItemString
 PyMapping_Size
 PyMapping_Values
+PyMarshal_ReadObjectFromString
+PyMarshal_WriteLongToFile
+PyMarshal_WriteObjectToFile
+PyMarshal_WriteObjectToString
 PyMem_Calloc
 PyMem_Free
 PyMem_Malloc
 PyMem_Realloc
 PyMemberDescr_Type
+PyMember_GetOne
+PyMember_SetOne
 PyMemoryView_FromMemory
 PyMemoryView_FromObject
 PyMemoryView_GetContiguous


I guess that you should not be added until this issue is solved.
msg392222 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-28 15:37
I hope the PR fixes that. I plan to merge tomorrow if there ar no objections to it.

On April 28, 2021 5:15:19 PM GMT+02:00, STINNER Victor <report@bugs.python.org> wrote:
>
>STINNER Victor <vstinner@python.org> added the comment:
>
>Right now, running "make regen-limited-abi" adds again functions which
>were removed from Doc/data/stable_abi.dat:
>https://github.com/python/cpython/pull/25687#issuecomment-828520575
>
>diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat
>index cdc7160250..6fe61743d6 100644
>--- a/Doc/data/stable_abi.dat
>+++ b/Doc/data/stable_abi.dat
>@@ -351,11 +351,17 @@ PyMapping_Length
> PyMapping_SetItemString
> PyMapping_Size
> PyMapping_Values
>+PyMarshal_ReadObjectFromString
>+PyMarshal_WriteLongToFile
>+PyMarshal_WriteObjectToFile
>+PyMarshal_WriteObjectToString
> PyMem_Calloc
> PyMem_Free
> PyMem_Malloc
> PyMem_Realloc
> PyMemberDescr_Type
>+PyMember_GetOne
>+PyMember_SetOne
> PyMemoryView_FromMemory
> PyMemoryView_FromObject
> PyMemoryView_GetContiguous
>
>
>I guess that you should not be added until this issue is solved.
>
>----------
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><https://bugs.python.org/issue43795>
>_______________________________________
msg392296 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-04-29 09:33
New changeset f6ee4dad589c0953283dacb577a2d808fda7aae9 by Petr Viktorin in branch 'master':
bpo-43795: Generate python3dll.c and doc data from manifest (PEP 652) (GH-25315)
https://github.com/python/cpython/commit/f6ee4dad589c0953283dacb577a2d808fda7aae9
msg393031 - (view) Author: miss-islington (miss-islington) Date: 2021-05-05 18:15
New changeset 8f3ef457ad286e73f272dadec8b98d2ea3a6ec72 by Miss Islington (bot) in branch '3.10':
bpo-43795: Mark PyCodec_Unregister as a function, not data, in stable ABI (GH-25920)
https://github.com/python/cpython/commit/8f3ef457ad286e73f272dadec8b98d2ea3a6ec72
msg393484 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-12 03:31
Unfortunately commit b05955d6f5f149523b5855a335444b7c6324bdb7 has broken all refleak buildbots:

https://buildbot.python.org/all/#/builders/259/builds/25/steps/5/logs/stdio

See example:

----------------------------------------------------------------------
Ran 53 tests in 5.629s
OK (skipped=1)
......
test_tools leaked [251, 251, 251] references, sum=753
test_tools leaked [91, 91, 91] memory blocks, sum=273
1 test failed again:
    test_tools

Per our buildbot policy, it will be reverted if is not fixed in 24h to avoid masking other errors.
msg393496 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-05-12 09:58
Thanks for the note!

This is due to `csv`. Adding a script in Tools/scripts that imports csv will also make the refleak check fail.
I'll investigate further.
msg393509 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-05-12 13:36
I opened https://bugs.python.org/issue44116 for the leak when trying to unload _csv.
Looks like investigating that will take a while, so I'll send a small PR to unblock the buildbots.
msg393624 - (view) Author: Carol Willing (willingc) * (Python committer) Date: 2021-05-14 05:29
New changeset 373937182ee029c282bea58cdda57ab41990f404 by Miss Islington (bot) in branch '3.10':
bpo-43795: PEP 652 user documentation (GH-25668) (GH-26034)
https://github.com/python/cpython/commit/373937182ee029c282bea58cdda57ab41990f404
msg394335 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-05-25 11:59
New changeset 68b9c20a98887bf0a92698f8df9f28b9440970dc by Miss Islington (bot) in branch '3.10':
bpo-43795: Remove Py_FrozenMain from the Limited API & Stable ABI  (GH-26241) (GH-26353)
https://github.com/python/cpython/commit/68b9c20a98887bf0a92698f8df9f28b9440970dc
msg395335 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-08 15:20
New changeset 257e400a19b34c7da6e2aa500d80b54e4c4dbf6f by Petr Viktorin in branch 'main':
bpo-43795: Note Stable ABI PEP in What's New (GH-26479)
https://github.com/python/cpython/commit/257e400a19b34c7da6e2aa500d80b54e4c4dbf6f
msg395337 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-08 16:00
New changeset 75185561a9a3b6dede3ad87bd83bab66847bd425 by Miss Islington (bot) in branch '3.10':
bpo-43795: Note Stable ABI PEP in What's New (GH-26479) (GH-26603)
https://github.com/python/cpython/commit/75185561a9a3b6dede3ad87bd83bab66847bd425
msg395913 - (view) Author: Petr Viktorin (petr.viktorin) * (Python committer) Date: 2021-06-16 09:22
New changeset 7cad9cb51bdae2144cbab330f13a607ba3471742 by Petr Viktorin in branch 'main':
bpo-43795: Don't list private names in the limited API (GH-26740)
https://github.com/python/cpython/commit/7cad9cb51bdae2144cbab330f13a607ba3471742
msg395915 - (view) Author: miss-islington (miss-islington) Date: 2021-06-16 09:53
New changeset 7fd40101b7130016fc98f05ce34746fed68ab542 by Miss Islington (bot) in branch '3.10':
bpo-43795: Don't list private names in the limited API (GH-26740)
https://github.com/python/cpython/commit/7fd40101b7130016fc98f05ce34746fed68ab542
History
Date User Action Args
2021-06-16 09:53:24miss-islingtonsetmessages: + msg395915
2021-06-16 09:22:53petr.viktorinsetmessages: + msg395913
2021-06-16 09:22:52miss-islingtonsetpull_requests: + pull_request25338
2021-06-15 14:52:31petr.viktorinsetpull_requests: + pull_request25326
2021-06-08 16:00:40petr.viktorinsetmessages: + msg395337
2021-06-08 15:20:17miss-islingtonsetpull_requests: + pull_request25185
2021-06-08 15:20:14petr.viktorinsetmessages: + msg395335
2021-06-02 15:17:03corona10setnosy: + corona10
2021-06-01 14:06:38petr.viktorinsetpull_requests: + pull_request25074
2021-05-25 12:29:47petr.viktorinsetpull_requests: + pull_request24946
2021-05-25 11:59:26pablogsalsetmessages: + msg394335
2021-05-25 11:42:11miss-islingtonsetpull_requests: + pull_request24943
2021-05-19 13:31:12petr.viktorinsetpull_requests: + pull_request24856
2021-05-14 09:34:12miss-islingtonsetpull_requests: + pull_request24764
2021-05-14 05:29:13willingcsetnosy: + willingc
messages: + msg393624
2021-05-13 14:59:38petr.viktorinsetpull_requests: + pull_request24742
2021-05-12 13:36:38petr.viktorinsetmessages: + msg393509
2021-05-12 09:58:23petr.viktorinsetmessages: + msg393496
2021-05-12 03:31:29pablogsalsetnosy: + pablogsal
messages: + msg393484
2021-05-11 14:04:42miss-islingtonsetpull_requests: + pull_request24681
2021-05-05 18:15:33miss-islingtonsetmessages: + msg393031
2021-05-05 17:32:29miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request24592
2021-05-05 16:32:35petr.viktorinsetpull_requests: + pull_request24588
2021-04-29 11:59:17orsenthilsetnosy: + orsenthil
pull_requests: + pull_request24407
2021-04-29 09:33:23petr.viktorinsetmessages: + msg392296
2021-04-28 15:37:58petr.viktorinsetmessages: + msg392222
2021-04-28 15:15:19vstinnersetmessages: + msg392212
2021-04-27 19:21:17petr.viktorinsetpull_requests: + pull_request24357
2021-04-23 12:18:02petr.viktorinsetmessages: + msg391686
2021-04-23 12:14:07petr.viktorinsetmessages: + msg391684
2021-04-20 14:30:44petr.viktorinsetpull_requests: + pull_request24208
2021-04-20 13:52:18petr.viktorinsetmessages: + msg391436
2021-04-20 13:49:30erlendaaslandsetmessages: + msg391435
2021-04-20 13:21:39petr.viktorinsetpull_requests: + pull_request24207
2021-04-20 12:07:25petr.viktorinsetmessages: + msg391432
2021-04-16 21:02:09petr.viktorinsetmessages: + msg391258
2021-04-16 17:23:35vstinnersetmessages: + msg391235
2021-04-16 16:08:19petr.viktorinsetnosy: + vstinner
messages: + msg391223
2021-04-16 15:25:38petr.viktorinsetmessages: + msg391217
2021-04-14 14:38:37petr.viktorinsetmessages: + msg391081
2021-04-10 20:26:50erlendaaslandsetnosy: + erlendaasland
2021-04-09 16:11:06petr.viktorinsetpull_requests: + pull_request24050
2021-04-09 15:47:40petr.viktorinsetkeywords: + patch
stage: patch review
pull_requests: + pull_request24047
2021-04-09 15:43:50petr.viktorincreate