This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Move PyArena C API to the internal C API
Type: Stage: resolved
Components: C API Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2021-02-17 18:31 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24688 closed shihai1991, 2021-03-01 17:41
PR 24907 merged vstinner, 2021-03-17 21:36
PR 24908 merged vstinner, 2021-03-17 22:18
PR 24910 merged vstinner, 2021-03-17 23:09
PR 24911 merged vstinner, 2021-03-17 23:38
PR 24912 merged vstinner, 2021-03-18 00:22
PR 24913 merged vstinner, 2021-03-18 02:23
PR 24914 closed vstinner, 2021-03-18 08:11
PR 24933 merged vstinner, 2021-03-19 12:35
PR 25000 merged vstinner, 2021-03-23 19:58
PR 25001 merged vstinner, 2021-03-23 20:28
PR 25007 merged vstinner, 2021-03-24 00:43
PR 25242 closed vstinner, 2021-04-07 09:10
PR 25243 merged vstinner, 2021-04-07 09:36
PR 25252 merged vstinner, 2021-04-07 11:53
PR 25256 closed vstinner, 2021-04-07 15:47
Messages (19)
msg387180 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-02-17 18:31
Include/cpython/pyarena.h declares 4 functions which are implemented in Python/pyarena.c. The header file exists to access these functions from the compiler. IMO there was no intent to expose these functions to the public Python C API.

Moreover, the PEP 384 explicitly excludes this API from the stable ABI (from the limited C API).

I propose to move these functions to the internal C API.

Since these functions are no even documented, I don't think that a deprecation period is needed.

Note: The PR 24550 moved Include/pyarena.h to Include/cpython/pyarena.h.
msg388973 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-17 22:11
New changeset b4536e1c6abe4c6219177a89e16575d05ea22f64 by Victor Stinner in branch 'master':
bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907)
https://github.com/python/cpython/commit/b4536e1c6abe4c6219177a89e16575d05ea22f64
msg388976 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-17 22:51
New changeset 526fdeb2278b61653df704d7cfcaedde504dee48 by Victor Stinner in branch 'master':
bpo-43244: Add pycore_ast.h header file (GH-24908)
https://github.com/python/cpython/commit/526fdeb2278b61653df704d7cfcaedde504dee48
msg388981 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 01:46
New changeset e0bf70d08c4a4a68782702e747e6bf7670667591 by Victor Stinner in branch 'master':
bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
https://github.com/python/cpython/commit/e0bf70d08c4a4a68782702e747e6bf7670667591
msg389001 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 08:54
New changeset 6af528b4ab342805534c0bfe61d84ed7bb519468 by Victor Stinner in branch 'master':
bpo-43244: Fix test_peg_generators on Windows (GH-24913)
https://github.com/python/cpython/commit/6af528b4ab342805534c0bfe61d84ed7bb519468
msg389016 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 13:57
New changeset eec8e61992fb654d4cf58de4d727c18622b8303e by Victor Stinner in branch 'master':
bpo-43244: Remove the PyAST_Validate() function (GH-24911)
https://github.com/python/cpython/commit/eec8e61992fb654d4cf58de4d727c18622b8303e
msg389018 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-18 14:33
The work on this issue started in Python 3.9 with bpo-21120 which excluded Python-ast.h, ast.h and asdl.h from the limited C API: commit 421a72af4deaec96a49a79951b9c2546a2faa13d.
msg389075 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-19 11:42
New changeset 28ad12f8fe889a741661eb99daacebd9243cc1ba by Victor Stinner in branch 'master':
bpo-43244: Remove symtable.h header file (GH-24910)
https://github.com/python/cpython/commit/28ad12f8fe889a741661eb99daacebd9243cc1ba
msg389408 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-23 19:47
New changeset 94faa0724f8cbae6867c491c8e465e35f4fdbfbb by Victor Stinner in branch 'master':
bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
https://github.com/python/cpython/commit/94faa0724f8cbae6867c491c8e465e35f4fdbfbb
msg389416 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-23 23:51
New changeset a81fca6ec8e0f748f8eafa12fb12cf9e12df465c by Victor Stinner in branch 'master':
bpo-43244: Add pycore_compile.h header file (GH-25000)
https://github.com/python/cpython/commit/a81fca6ec8e0f748f8eafa12fb12cf9e12df465c
msg389419 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-24 00:29
New changeset 57364ce34e0492fbc8b0a6b8c882f384bb489457 by Victor Stinner in branch 'master':
bpo-43244: Remove parser_interface.h header file (GH-25001)
https://github.com/python/cpython/commit/57364ce34e0492fbc8b0a6b8c882f384bb489457
msg389423 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-24 01:23
New changeset 8370e07e1e5b626e78ddc7aadbfaf248976c4454 by Victor Stinner in branch 'master':
bpo-43244: Remove the pyarena.h header (GH-25007)
https://github.com/python/cpython/commit/8370e07e1e5b626e78ddc7aadbfaf248976c4454
msg389424 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-24 01:26
Ok, the whole C API related to parser, compiler, AST, ASDL and PyArena moved to the internal C API. So we will have more freedom to evolve it!
msg389501 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-25 13:56
In short, this issue is a follow-up of bpo-40939.

These C API removal are related to the final step of the PEP 617 "New PEG parser for CPython": removal of the PyParser C API.

* The PyParser C API produced "node*" objects, like PyParser_SimpleParseFile(FILE*) => node*. This C API was removed in Python 3.10: bpo-40939 "Remove the old parser".

* The AST C API takes "node*" as input: PyAST_FromNode(node*) => mod_ty. I removed these APIs.

* symtable takes "mod_ty" as input.

* ASDL and PyArena API were only useful for the removed AST C APIs.

I searched for removed functions in the top PyPI 4000 projects, there is a single project on 4000: typed_ast. This typed_ast project is special, it copies directly code from CPython. Even after the C API removed, typed_ast can continue to use the internal C API, it only has to define the Py_BUILD_CORE_MODULE macro.
msg390416 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 11:01
New changeset d36d6a9c1808e87628ebaa855d4bec80130189f4 by Victor Stinner in branch 'master':
bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)
https://github.com/python/cpython/commit/d36d6a9c1808e87628ebaa855d4bec80130189f4
msg390444 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 16:24
typed-ast was updated for these changes and to avoid including explicitly pydebug.h (it's included by Python.h: see bpo-35134 change): https://github.com/python/typed_ast/pull/158
msg390472 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 19:34
New changeset d27f8d2e07d31670af469ef387a37bc9e96ea8ad by Victor Stinner in branch 'master':
bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() (GH-25252)
https://github.com/python/cpython/commit/d27f8d2e07d31670af469ef387a37bc9e96ea8ad
msg390473 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-04-07 19:35
"Remove Yield macro from pycore_ast.h"

I'm happy that this one is gone, it was a common and annoying compiler warning on Windows!
msg394489 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-26 22:44
I marked bpo-35197 as a duplicate of this issue.
History
Date User Action Args
2022-04-11 14:59:41adminsetgithub: 87410
2021-05-26 22:44:01vstinnersetmessages: + msg394489
2021-05-26 22:43:50vstinnerlinkissue35197 superseder
2021-04-07 19:35:41vstinnersetmessages: + msg390473
2021-04-07 19:34:32vstinnersetmessages: + msg390472
2021-04-07 19:33:06vstinnerlinkissue43767 superseder
2021-04-07 16:24:32vstinnersetmessages: + msg390444
2021-04-07 15:47:22vstinnersetpull_requests: + pull_request23992
2021-04-07 11:53:22vstinnersetpull_requests: + pull_request23990
2021-04-07 11:01:17vstinnersetmessages: + msg390416
2021-04-07 09:36:09vstinnersetpull_requests: + pull_request23983
2021-04-07 09:10:07vstinnersetpull_requests: + pull_request23982
2021-03-25 13:56:23vstinnersetmessages: + msg389501
2021-03-24 01:26:54vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg389424

stage: patch review -> resolved
2021-03-24 01:23:08vstinnersetmessages: + msg389423
2021-03-24 00:43:47vstinnersetpull_requests: + pull_request23765
2021-03-24 00:29:17vstinnersetmessages: + msg389419
2021-03-23 23:51:57vstinnersetmessages: + msg389416
2021-03-23 20:28:04vstinnersetpull_requests: + pull_request23759
2021-03-23 19:58:39vstinnersetpull_requests: + pull_request23758
2021-03-23 19:47:49vstinnersetmessages: + msg389408
2021-03-19 12:35:01vstinnersetpull_requests: + pull_request23694
2021-03-19 11:42:09vstinnersetmessages: + msg389075
2021-03-18 14:33:56vstinnersetmessages: + msg389018
2021-03-18 13:57:56vstinnersetmessages: + msg389016
2021-03-18 08:54:25vstinnersetmessages: + msg389001
2021-03-18 08:11:19vstinnersetpull_requests: + pull_request23677
2021-03-18 02:23:00vstinnersetpull_requests: + pull_request23676
2021-03-18 01:46:30vstinnersetmessages: + msg388981
2021-03-18 00:22:37vstinnersetpull_requests: + pull_request23675
2021-03-17 23:38:04vstinnersetpull_requests: + pull_request23673
2021-03-17 23:09:44vstinnersetpull_requests: + pull_request23672
2021-03-17 22:51:09vstinnersetmessages: + msg388976
2021-03-17 22:18:02vstinnersetpull_requests: + pull_request23670
2021-03-17 22:11:10vstinnersetmessages: + msg388973
2021-03-17 21:36:07vstinnersetpull_requests: + pull_request23669
2021-03-01 17:41:48shihai1991setkeywords: + patch
nosy: + shihai1991

pull_requests: + pull_request23470
stage: patch review
2021-02-17 18:31:30vstinnercreate