classification
Title: Make lib2to3 grammar better match Python, support the := walrus
Type: behavior Stage: needs patch
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Peter Ludemann, benjamin.peterson, fireattack, georg.brandl, gregory.p.smith, lisroach, lukasz.langa, miss-islington, pablogsal, thatch
Priority: normal Keywords: patch

Created on 2019-04-06 01:41 by thatch, last changed 2020-10-19 23:41 by gregory.p.smith.

Pull Requests
URL Status Linked Edit
PR 12702 merged python-dev, 2019-04-06 01:42
PR 12703 closed thatch, 2019-04-06 01:55
PR 19315 merged miss-islington, 2020-04-02 22:36
PR 19317 merged thatch, 2020-04-02 23:43
Messages (12)
msg339522 - (view) Author: Tim Hatch (thatch) * Date: 2019-04-06 01:41
The grammar in lib2to3 is out of date and can't parse `:=` nor `f(**not x)` from running on real code.  I've done a cursory `diff -uw Grammar/Grammar Lib/lib2to3/grammar.txt`, and would like to fix lib2to3 so we can merge into both fissix and blib2to3, to avoid further divergence of the forks.

I'm unsure if I need a separate bug per pull request, but need at least one to get started.
msg339669 - (view) Author: Tim Hatch (thatch) * Date: 2019-04-08 19:49
jreese reminded me of pep570, which will make more grammar changes.  I'm open to the idea of replacing the grammar with the live one, plus porting the 2isms forward like print, eval, except with comma.

My sincere hope is that everyone that depends on this structure will have tests (mine and lib2to3 do); the only big user I'm aware of is probably libfuturize.  Definitely worth a changelog entry if this is the way forward.
msg339796 - (view) Author: Tim Hatch (thatch) * Date: 2019-04-09 18:06
Here's approximately what it would look like to do the big change now: https://github.com/python/cpython/compare/master...thatch:lib2to3-update-grammar (one test failing, and some helpers may need more test coverage)
msg340791 - (view) Author: Lisa Roach (lisroach) * (Python committer) Date: 2019-04-24 16:47
I agree we should get lib2to3 up to date.

Looks like for *args and **kwargs there is issue33348 (this has a PR) and issue32496 (no PR) and related closed issue24791 and issue24176. 

Adding `:=` seems straighforward to me, as for the big change maybe @benjamin.peterson would be interested in commenting?
msg340802 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-04-24 19:37
For the changes of PEP570, please wait until I merge the implementation to do the grammar changes in lib2to3 for that.
msg340848 - (view) Author: Tim Hatch (thatch) * Date: 2019-04-25 16:38
My strong preference would be getting the lib2to3 grammar to be the python grammar + additions, to make future changes easier to merge.  The strongest argument against doing that is the backwards-incompatibility of patterns -- some won't compile, while others will compile but do something unexpected).

It's good to hear (or at least infer) that parsing modern code is also a goal of lib2to3.
msg355108 - (view) Author: Peter Ludemann (Peter Ludemann) Date: 2019-10-21 22:34
Re: breakage due to changes in structure (https://bugs.python.org/issue36541#msg339669) ... this has already happened in the past (e.g., type annotations and async). 

It's probably a good idea to add some documentation that structure changes can be expected with each release of Python.
msg355252 - (view) Author: Peter Ludemann (Peter Ludemann) Date: 2019-10-23 19:05
Also the Grammar.txt diffs look about the same size as I've seen with other upgrades to lib2to3 when the Python grammar changed.
msg365716 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-03 19:14
New changeset 96c5f5a3a3fabf43e8114d0dbc30bed409da1ba6 by Tim Hatch in branch '3.7':
[3.7] bpo-36541: lib2to3: Support named assignment expressions (GH-12702) (GH-19317)
https://github.com/python/cpython/commit/96c5f5a3a3fabf43e8114d0dbc30bed409da1ba6
msg365717 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-03 19:18
master/3.9 changeset:
https://github.com/python/cpython/commit/3c3aa4516c70753de06bb142b6793d01330fcf0f

3.8 changeset: https://github.com/python/cpython/commit/1098671e4e5ec1513247f05598158eaa3428c5be
msg365718 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-04-03 19:18
Support for `:=` is in, are we still lacking `f(**not x)` support?
msg379062 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2020-10-19 23:41
Parsing support for `f(**mapping)` support is indeed still missing.

as lib2to3 is pending deprecation at this point, i'm not going to work on this.  anyone is welcome to pick it up.  modifying the lib2to3 grammar, and any related code, and adding a test is what's needed to parse that syntax.
History
Date User Action Args
2020-10-19 23:41:24gregory.p.smithsetassignee: gregory.p.smith ->
stage: patch review -> needs patch
messages: + msg379062
versions: + Python 3.10, - Python 3.7
2020-04-03 19:18:58gregory.p.smithsetmessages: + msg365718
2020-04-03 19:18:17gregory.p.smithsetmessages: + msg365717
2020-04-03 19:14:18gregory.p.smithsetmessages: + msg365716
2020-04-02 23:43:31thatchsetpull_requests: + pull_request18682
2020-04-02 22:36:27miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request18680
2020-04-02 02:09:45fireattacksetnosy: + fireattack
2019-10-24 06:04:11gregory.p.smithsettitle: Make lib2to3 grammar more closely match Python -> Make lib2to3 grammar better match Python, support the := walrus
2019-10-24 06:01:55gregory.p.smithsetassignee: gregory.p.smith

nosy: + gregory.p.smith
2019-10-24 05:49:24gregory.p.smithsetversions: + Python 3.9, - Python 3.6
2019-10-23 19:05:53Peter Ludemannsetmessages: + msg355252
2019-10-21 22:34:06Peter Ludemannsetnosy: + Peter Ludemann
messages: + msg355108
2019-06-15 12:16:09xtreaklinkissue37248 superseder
2019-04-25 16:38:46thatchsetmessages: + msg340848
2019-04-24 19:37:22pablogsalsetmessages: + msg340802
2019-04-24 16:47:01lisroachsetnosy: + lisroach
messages: + msg340791
2019-04-09 23:17:27xtreaksetnosy: + benjamin.peterson
2019-04-09 18:06:15thatchsetmessages: + msg339796
2019-04-08 19:49:45thatchsetmessages: + msg339669
2019-04-06 18:46:55pablogsalsetnosy: + pablogsal
2019-04-06 01:55:27thatchsetpull_requests: + pull_request12627
2019-04-06 01:42:20python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request12626
2019-04-06 01:41:20thatchcreate