classification
Title: Merge typed_ast back into CPython
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, Ethan Smith, gregory.p.smith, gvanrossum, levkivskyi, lukasz.langa
Priority: normal Keywords: patch, patch, patch

Created on 2019-01-18 00:33 by gvanrossum, last changed 2019-02-12 00:23 by gvanrossum. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 11645 closed gvanrossum, 2019-01-22 02:26
PR 11645 closed gvanrossum, 2019-01-22 02:26
PR 11645 closed gvanrossum, 2019-01-22 02:26
PR 11714 merged gvanrossum, 2019-01-31 17:05
PR 11714 merged gvanrossum, 2019-01-31 17:05
PR 11714 merged gvanrossum, 2019-01-31 17:05
PR 11766 merged gvanrossum, 2019-02-06 00:36
PR 11766 merged gvanrossum, 2019-02-06 00:36
PR 11766 merged gvanrossum, 2019-02-06 00:36
Messages (11)
msg333919 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-18 00:33
(This started at https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377. It's somewhat related to https://bugs.python.org/issue33337.) 

I now have a thorough understanding of what typed_ast does, and I think it would be straightforward to port it upstream. We’d need to define two new tokens to represent `# type: ignore` and `# type: <whatever>`, and tokenizer code to recognize these. Then we need a new flag to be passed to the tokenizer (via the parser) that enables this behavior. We make a small number of changes to `Grammar` (inserting optional `TYPE_COMMENT` tokens and to `Python.asdl` (adding fields to a few node types to hold the optional type comment), and a fair number of changes to `ast.c` to extract the type comments. We have similar patches for 3.6 and 3.7, so it's a simple matter of porting those patches to 3.8.

By default, `ast.parse()` should not return type comments, since this would reject some perfectly good Python code (with sonething looking like a type comment in a place where the grammar doesn’t allow it). But passing an new flag will cause the tokenizer to process type comments and the returned tree will contain them.

I could produce a PR with this in a few days (having just gone over most of the process for porting typed_ast from 3.6 to 3.7).

There’s one more feature I’d like to lobby for – a feature_version flag that modifies the grammar slightly so it resembles an older version of Python (going back to 3.4). This is used in mypy to decouple the Python version you’re running from the Python version for which you’re checking compatibility (useful when checking code that will be deployed on a system with a different Python version installed). I imagine this would be useful to other linters as well, and the implementation is mostly manipulating whether `async` and `await` are keywords. But if there’s pushback to this part I can live without it – the rest of the work is still useful.

In the next few days I will produce a PR so people can see for themselves.

In https://discuss.python.org/t/merge-typed-ast-back-into-cpython/377/17, Łukasz offered to merge my PR.
msg334068 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-01-19 21:15
Seems also related to https://bugs.python.org/issue24119

with python2 / python3.5 (hopefully) rapidly falling off in usage I would assume the specialized treatment of `# type: ...` comments would become less and less necessary

Though I guess there's still `# type: ignore`, though if I recall correctly ignores are bubbled up to the module level and are applied on a line-by-line basis and wouldn't necessarily need specialized AST treatment (though a second parse over the token stream is a bit unfortunate)
msg334071 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-19 21:57
You’d be surprised how tenacious old versions are. 

Anywa, I am working on this on my copious spare time — you can follow my progress at https://github.com/gvanrossum/cpython/tree/ast-type-comments?files=1 .
msg334073 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2019-01-19 22:17
> You’d be surprised how tenacious old versions are. 

heh that's true, at my last job we finally got rid of python2.6 in 2016 :'(

anyway -- I don't mean to discourage this, definitely seems valuable to the maintenance of mypy!
msg334371 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-25 22:26
The PR is ready for reviews now.
msg334500 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-29 03:10
Also the tests now all pass; for now I am happy with the solution I found for the indentation error (see https://github.com/python/cpython/pull/11645#issuecomment-456627216).
msg334624 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2019-01-31 11:40
New changeset dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c by Łukasz Langa (Guido van Rossum) in branch 'master':
bpo-35766: Merge typed_ast back into CPython (GH-11645)
https://github.com/python/cpython/commit/dcfcd146f8e6fc5c2fc16a4c192a0c5f5ca8c53c
msg334714 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-01 19:37
New changeset 3a32e3bf880f0842ac73d18285f5a1caa902a2ca by Guido van Rossum in branch 'master':
bpo-35766 follow-up: Kill half-support for FunctionType in PyAST_obj2mod (#11714)
https://github.com/python/cpython/commit/3a32e3bf880f0842ac73d18285f5a1caa902a2ca
msg335234 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-11 16:10
New changeset 4b250fc1da9c893803cf724a4974450b5e10bd8a by Guido van Rossum in branch 'master':
bpo-35766 follow-up: Add an error check to new_type_comment() (#11766)
https://github.com/python/cpython/commit/4b250fc1da9c893803cf724a4974450b5e10bd8a
msg335276 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-12 00:11
I have some follow-up wishes but I'll create a new issue.
msg335278 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-02-12 00:23
See https://bugs.python.org/issue35975
History
Date User Action Args
2019-02-12 00:23:54gvanrossumsetkeywords: patch, patch, patch

messages: + msg335278
2019-02-12 00:11:24gvanrossumsetstatus: open -> closed
messages: + msg335276

keywords: patch, patch, patch
resolution: fixed
stage: patch review -> resolved
2019-02-11 16:10:47gvanrossumsetmessages: + msg335234
2019-02-06 00:36:39gvanrossumsetpull_requests: + pull_request11725
2019-02-06 00:36:28gvanrossumsetpull_requests: + pull_request11724
2019-02-06 00:36:15gvanrossumsetpull_requests: + pull_request11723
2019-02-01 19:37:48gvanrossumsetmessages: + msg334714
2019-01-31 17:06:03gvanrossumsetpull_requests: + pull_request11575
2019-01-31 17:05:49gvanrossumsetpull_requests: + pull_request11574
2019-01-31 17:05:35gvanrossumsetpull_requests: + pull_request11573
2019-01-31 11:40:34lukasz.langasetnosy: + lukasz.langa
messages: + msg334624
2019-01-29 03:10:17gvanrossumsetkeywords: patch, patch, patch

messages: + msg334500
2019-01-25 22:26:54gvanrossumsetkeywords: patch, patch, patch

messages: + msg334371
2019-01-23 05:55:42Ethan Smithsetnosy: + Ethan Smith
2019-01-22 02:26:21gvanrossumsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request11429
2019-01-22 02:26:14gvanrossumsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11428
2019-01-22 02:26:07gvanrossumsetkeywords: + patch
stage: needs patch -> needs patch
pull_requests: + pull_request11427
2019-01-21 06:06:21gregory.p.smithsetnosy: + gregory.p.smith
2019-01-19 22:17:50Anthony Sottilesetmessages: + msg334073
2019-01-19 21:57:16gvanrossumsetmessages: + msg334071
2019-01-19 21:15:42Anthony Sottilesetmessages: + msg334068
2019-01-19 21:12:08Anthony Sottilesetnosy: + Anthony Sottile
2019-01-19 12:11:17levkivskyisetnosy: + levkivskyi
2019-01-18 00:33:01gvanrossumcreate