Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merge typed_ast back into CPython #79947

Closed
gvanrossum opened this issue Jan 18, 2019 · 28 comments
Closed

Merge typed_ast back into CPython #79947

gvanrossum opened this issue Jan 18, 2019 · 28 comments
Labels
3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@gvanrossum
Copy link
Member

BPO 35766
Nosy @gvanrossum, @gpshead, @vstinner, @ambv, @ilevkivskyi, @asottile, @ethanhs, @miss-islington
PRs
  • bpo-35766: Merge typed_ast back into CPython #11645
  • bpo-35766: Merge typed_ast back into CPython #11645
  • bpo-35766: Merge typed_ast back into CPython #11645
  • bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714
  • bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714
  • bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod #11714
  • bpo-35766 follow-up: Add an error check to new_type_comment() #11766
  • bpo-35766 follow-up: Add an error check to new_type_comment() #11766
  • bpo-35766 follow-up: Add an error check to new_type_comment() #11766
  • bpo-35766: Whats new in the ast and typing modules #13984
  • [3.8] bpo-35766: What's new in the ast and typing modules (GH-13984) #13987
  • bpo-35766: Change format for feature_version to (major, minor) #13992
  • [3.8] bpo-35766: Change format for feature_version to (major, minor) (GH-13992) #13993
  • bpo-35766: compile(): rename feature_version parameter #13994
  • [3.8] bpo-35766: compile(): rename feature_version parameter (GH-13994) #13995
  • [3.8] bpo-35766: compile(): rename feature_version parameter (GH-13994) #14001
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2019-02-12.00:11:24.565>
created_at = <Date 2019-01-18.00:33:01.067>
labels = ['3.8', 'type-feature', 'library']
title = 'Merge typed_ast back into CPython'
updated_at = <Date 2019-06-12.15:34:26.524>
user = 'https://github.com/gvanrossum'

    bugs.python.org fields:

    activity = <Date 2019-06-12.15:34:26.524>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-02-12.00:11:24.565>
closer = 'gvanrossum'
components = ['Library (Lib)']
creation = <Date 2019-01-18.00:33:01.067>
creator = 'gvanrossum'
dependencies = []
files = []
hgrepos = []
issue_num = 35766
keywords = ['patch', 'patch', 'patch']
message_count = 28.0
messages = ['333919', '334068', '334071', '334073', '334371', '334500', '334624', '334714', '335234', '335276', '335278', '345245', '345248', '345249', '345256', '345264', '345265', '345267', '345270', '345273', '345285', '345286', '345292', '345293', '345294', '345364', '345366', '345372']
nosy_count = 8.0
nosy_names = ['gvanrossum', 'gregory.p.smith', 'vstinner', 'lukasz.langa', 'levkivskyi', 'Anthony Sottile', 'ethan smith', 'miss-islington']
pr_nums = ['11645', '11645', '11645', '11714', '11714', '11714', '11766', '11766', '11766', '13984', '13987', '13992', '13993', '13994', '13995', '14001']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35766'
versions = ['Python 3.8']
    @gvanrossum
    Copy link
    Member Author

    (This started at https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's somewhat related to https://bugs.python.org/issue33337.)

    I now have a thorough understanding of what typed_ast does, and I think it would be straightforward to port it upstream. We’d need to define two new tokens to represent # type: ignore and # type: <whatever>, and tokenizer code to recognize these. Then we need a new flag to be passed to the tokenizer (via the parser) that enables this behavior. We make a small number of changes to Grammar (inserting optional TYPE_COMMENT tokens and to Python.asdl (adding fields to a few node types to hold the optional type comment), and a fair number of changes to ast.c to extract the type comments. We have similar patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 3.8.

    By default, ast.parse() should not return type comments, since this would reject some perfectly good Python code (with sonething looking like a type comment in a place where the grammar doesn’t allow it). But passing an new flag will cause the tokenizer to process type comments and the returned tree will contain them.

    I could produce a PR with this in a few days (having just gone over most of the process for porting typed_ast from 3.6 to 3.7).

    There’s one more feature I’d like to lobby for – a feature_version flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether async and await are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

    In the next few days I will produce a PR so people can see for themselves.

    In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, Łukasz offered to merge my PR.

    @gvanrossum gvanrossum added 3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Jan 18, 2019
    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Jan 19, 2019

    Seems also related to https://bugs.python.org/issue24119

    with python2 / python3.5 (hopefully) rapidly falling off in usage I would assume the specialized treatment of # type: ... comments would become less and less necessary

    Though I guess there's still # type: ignore, though if I recall correctly ignores are bubbled up to the module level and are applied on a line-by-line basis and wouldn't necessarily need specialized AST treatment (though a second parse over the token stream is a bit unfortunate)

    @gvanrossum
    Copy link
    Member Author

    You’d be surprised how tenacious old versions are.

    Anywa, I am working on this on my copious spare time — you can follow my progress at https://github.com/gvanrossum/cpython/tree/ast-type-comments?files=1 .

    @asottile
    Copy link
    Mannequin

    asottile mannequin commented Jan 19, 2019

    You’d be surprised how tenacious old versions are.

    heh that's true, at my last job we finally got rid of python2.6 in 2016 :'(

    anyway -- I don't mean to discourage this, definitely seems valuable to the maintenance of mypy!

    @gvanrossum
    Copy link
    Member Author

    The PR is ready for reviews now.

    @gvanrossum
    Copy link
    Member Author

    Also the tests now all pass; for now I am happy with the solution I found for the indentation error (see #11645 (comment)).

    @ambv
    Copy link
    Contributor

    ambv commented Jan 31, 2019

    New changeset dcfcd14 by Łukasz Langa (Guido van Rossum) in branch 'master':
    bpo-35766: Merge typed_ast back into CPython (GH-11645)
    dcfcd14

    @gvanrossum
    Copy link
    Member Author

    New changeset 3a32e3b by Guido van Rossum in branch 'master':
    bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod (bpo-11714)
    3a32e3b

    @gvanrossum
    Copy link
    Member Author

    New changeset 4b250fc by Guido van Rossum in branch 'master':
    bpo-35766 follow-up: Add an error check to new_type_comment() (bpo-11766)
    4b250fc

    @gvanrossum
    Copy link
    Member Author

    I have some follow-up wishes but I'll create a new issue.

    @gvanrossum
    Copy link
    Member Author

    See https://bugs.python.org/issue35975

    @vstinner
    Copy link
    Member

    I don't see anything in What's New in Python 3.8?:
    https://docs.python.org/dev/whatsnew/3.8.html

    Would it be possible to document the change somewhere?

    @ilevkivskyi
    Copy link
    Member

    Would it be possible to document the change somewhere?

    Most of these things are in the docs for ast module.

    But indeed none of static typing related features have been added to What's New. I have an item on mypy todo list to add four typing PEPs plus new AST features (also adding end_lineno and end_col_offset) also there, but didn't have time last weekend.

    @ilevkivskyi
    Copy link
    Member

    Typo: "mypy todo list" should be "my todo list" :-)

    @gvanrossum
    Copy link
    Member Author

    I'll create a PR that updates the What's New docs with news about all of these.

    @vstinner
    Copy link
    Member

    feature_version=N allows specifying the minor version of an earlier Python 3 version. (For example, feature_version=4 will treat async and await as non-reserved words.)

    What will happen with Python 4? Why not using "full" Python version like (3, 4) or 0x304?

    @gvanrossum
    Copy link
    Member Author

    What will happen with Python 4? Why not using "full" Python version like (3, 4) or 0x304?

    I don't think Python 4 is just around the corner, so I prefer to kick that problem down the road. I'm sure we can come up with a sufficiently backwards-compatible API. (I'd prefer not to go the hex route, because printing the value would show as 52, which is pretty meaningless.)

    @gvanrossum
    Copy link
    Member Author

    New changeset 9b33ce4 by Guido van Rossum in branch 'master':
    bpo-35766: What's new in the ast and typing modules (bpo-13984)
    9b33ce4

    @miss-islington
    Copy link
    Contributor

    New changeset 7856888 by Miss Islington (bot) in branch '3.8':
    bpo-35766: What's new in the ast and typing modules (GH-13984)
    7856888

    @vstinner
    Copy link
    Member

    I'm sure we can come up with a sufficiently backwards-compatible API. (I'd prefer not to go the hex route, because printing the value would show as 52, which is pretty meaningless.)

    I propose to replace 4 with (3, 4) to be more future-proof. This version format is already used by sys.version_info for example.

    @gvanrossum
    Copy link
    Member Author

    But this is the format currently used by typed_ast. I think it would just cause a cascade of annoying failures for tools that switch between typed_ast and the 3.8 version of ast. Such as mypy.

    @vstinner
    Copy link
    Member

    But this is the format currently used by typed_ast. I think it would just cause a cascade of annoying failures for tools that switch between typed_ast and the 3.8 version of ast. Such as mypy.

    typed_ast is a 3rd party project. But here we are talking about Python stdlib. If a project moves from typed_ast to stdlib ast, IMHO it should be quite easy to replace 4 with (3, 4), especially if 4 raises an error, no?

    IMHO it's a good opportunity to fix it.

    I would prefer to not have to batch many backward incompatible changes into Python 4...

    @miss-islington
    Copy link
    Contributor

    New changeset 10b55c1 by Miss Islington (bot) (Guido van Rossum) in branch 'master':
    bpo-35766: Change format for feature_version to (major, minor) (GH-13992)
    10b55c1

    @vstinner
    Copy link
    Member

    New changeset efdf6ca by Victor Stinner in branch 'master':
    bpo-35766: compile(): rename feature_version parameter (GH-13994)
    efdf6ca

    @vstinner
    Copy link
    Member

    New changeset 3ba2107 by Victor Stinner (Miss Islington (bot)) in branch '3.8':
    bpo-35766: Change format for feature_version to (major, minor) (GH-13992) (GH-13993)
    3ba2107

    @vstinner
    Copy link
    Member

    New changeset b2fd32b by Victor Stinner in branch '3.8':
    bpo-35766: compile(): rename feature_version parameter (GH-13994) (GH-14001)
    b2fd32b

    @gvanrossum
    Copy link
    Member Author

    Thanks for taking care of this!

    --Guido (mobile)

    @vstinner
    Copy link
    Member

    I created bpo-37253: "PyCompilerFlags got a new cf_feature_version field".

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.8 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    5 participants
    @ambv @vstinner @gvanrossum @ilevkivskyi @miss-islington