Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move PyArena C API to the internal C API #87410

Closed
vstinner opened this issue Feb 17, 2021 · 19 comments
Closed

Move PyArena C API to the internal C API #87410

vstinner opened this issue Feb 17, 2021 · 19 comments
Labels
3.10 only security fixes topic-C-API

Comments

@vstinner
Copy link
Member

BPO 43244
Nosy @vstinner, @shihai1991
PRs
  • bpo-43244: Move pyarena.h from include/cpython to include/internal/. #24688
  • bpo-43244: Rename pycore_ast.h to pycore_ast_state.h #24907
  • bpo-43244: Add pycore_ast.h header file #24908
  • bpo-43244: Remove symtable.h header file #24910
  • bpo-43244: Remove PyAST_Validate() function #24911
  • bpo-43244: test_peg_generator defines _Py_TEST_PEGEN macro #24912
  • bpo-43244: Fix test_peg_generators on Windows #24913
  • bpo-43244: GitHub Action builds Python in debug mode on Windows #24914
  • bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers #24933
  • bpo-43244: Add pycore_compile.h header file #25000
  • bpo-43244: Remove parser_interface.h header file #25001
  • bpo-43244: Remove pyarena.h header #25007
  • bpo-43244: Fix compiler warning in compile.c about Yield #25242
  • bpo-43244: Remove Yield macro from pycore_ast.h #25243
  • bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() #25252
  • bpo-43244: Remove aliases from pycore_ast.h #25256
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-03-24.01:26:54.289>
created_at = <Date 2021-02-17.18:31:30.182>
labels = ['expert-C-API', '3.10']
title = 'Move PyArena C API to the internal C API'
updated_at = <Date 2021-05-26.22:44:01.883>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2021-05-26.22:44:01.883>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-03-24.01:26:54.289>
closer = 'vstinner'
components = ['C API']
creation = <Date 2021-02-17.18:31:30.182>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 43244
keywords = ['patch']
message_count = 19.0
messages = ['387180', '388973', '388976', '388981', '389001', '389016', '389018', '389075', '389408', '389416', '389419', '389423', '389424', '389501', '390416', '390444', '390472', '390473', '394489']
nosy_count = 2.0
nosy_names = ['vstinner', 'shihai1991']
pr_nums = ['24688', '24907', '24908', '24910', '24911', '24912', '24913', '24914', '24933', '25000', '25001', '25007', '25242', '25243', '25252', '25256']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue43244'
versions = ['Python 3.10']
    @vstinner
    Copy link
    Member Author

    Include/cpython/pyarena.h declares 4 functions which are implemented in Python/pyarena.c. The header file exists to access these functions from the compiler. IMO there was no intent to expose these functions to the public Python C API.

    Moreover, the PEP-384 explicitly excludes this API from the stable ABI (from the limited C API).

    I propose to move these functions to the internal C API.

    Since these functions are no even documented, I don't think that a deprecation period is needed.

    Note: The PR 24550 moved Include/pyarena.h to Include/cpython/pyarena.h.

    @vstinner vstinner added 3.10 only security fixes topic-C-API labels Feb 17, 2021
    @vstinner
    Copy link
    Member Author

    New changeset b4536e1 by Victor Stinner in branch 'master':
    bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (GH-24907)
    b4536e1

    @vstinner
    Copy link
    Member Author

    New changeset 526fdeb by Victor Stinner in branch 'master':
    bpo-43244: Add pycore_ast.h header file (GH-24908)
    526fdeb

    @vstinner
    Copy link
    Member Author

    New changeset e0bf70d by Victor Stinner in branch 'master':
    bpo-43244: Fix test_peg_generator for PyAST_Validate() (GH-24912)
    e0bf70d

    @vstinner
    Copy link
    Member Author

    New changeset 6af528b by Victor Stinner in branch 'master':
    bpo-43244: Fix test_peg_generators on Windows (GH-24913)
    6af528b

    @vstinner
    Copy link
    Member Author

    New changeset eec8e61 by Victor Stinner in branch 'master':
    bpo-43244: Remove the PyAST_Validate() function (GH-24911)
    eec8e61

    @vstinner
    Copy link
    Member Author

    The work on this issue started in Python 3.9 with bpo-21120 which excluded Python-ast.h, ast.h and asdl.h from the limited C API: commit 421a72a.

    @vstinner
    Copy link
    Member Author

    New changeset 28ad12f by Victor Stinner in branch 'master':
    bpo-43244: Remove symtable.h header file (GH-24910)
    28ad12f

    @vstinner
    Copy link
    Member Author

    New changeset 94faa07 by Victor Stinner in branch 'master':
    bpo-43244: Remove ast.h, asdl.h, Python-ast.h headers (GH-24933)
    94faa07

    @vstinner
    Copy link
    Member Author

    New changeset a81fca6 by Victor Stinner in branch 'master':
    bpo-43244: Add pycore_compile.h header file (GH-25000)
    a81fca6

    @vstinner
    Copy link
    Member Author

    New changeset 57364ce by Victor Stinner in branch 'master':
    bpo-43244: Remove parser_interface.h header file (GH-25001)
    57364ce

    @vstinner
    Copy link
    Member Author

    New changeset 8370e07 by Victor Stinner in branch 'master':
    bpo-43244: Remove the pyarena.h header (GH-25007)
    8370e07

    @vstinner
    Copy link
    Member Author

    Ok, the whole C API related to parser, compiler, AST, ASDL and PyArena moved to the internal C API. So we will have more freedom to evolve it!

    @vstinner
    Copy link
    Member Author

    In short, this issue is a follow-up of bpo-40939.

    These C API removal are related to the final step of the PEP-617 "New PEG parser for CPython": removal of the PyParser C API.

    • The PyParser C API produced "node*" objects, like PyParser_SimpleParseFile(FILE*) => node*. This C API was removed in Python 3.10: bpo-40939 "Remove the old parser".

    • The AST C API takes "node*" as input: PyAST_FromNode(node*) => mod_ty. I removed these APIs.

    • symtable takes "mod_ty" as input.

    • ASDL and PyArena API were only useful for the removed AST C APIs.

    I searched for removed functions in the top PyPI 4000 projects, there is a single project on 4000: typed_ast. This typed_ast project is special, it copies directly code from CPython. Even after the C API removed, typed_ast can continue to use the internal C API, it only has to define the Py_BUILD_CORE_MODULE macro.

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 7, 2021

    New changeset d36d6a9 by Victor Stinner in branch 'master':
    bpo-43244: Remove Yield macro from pycore_ast.h (GH-25243)
    d36d6a9

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 7, 2021

    typed-ast was updated for these changes and to avoid including explicitly pydebug.h (it's included by Python.h: see bpo-35134 change): python/typed_ast#158

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 7, 2021

    New changeset d27f8d2 by Victor Stinner in branch 'master':
    bpo-43244: Rename pycore_ast.h functions to _PyAST_xxx() (GH-25252)
    d27f8d2

    @vstinner
    Copy link
    Member Author

    vstinner commented Apr 7, 2021

    "Remove Yield macro from pycore_ast.h"

    I'm happy that this one is gone, it was a common and annoying compiler warning on Windows!

    @vstinner
    Copy link
    Member Author

    I marked bpo-35197 as a duplicate of this issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.10 only security fixes topic-C-API
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner