classification
Title: 2to3 parser failure caused by a comma after a generator expression
Type: crash Stage:
Components: 2to3 (2.x to 3.x conversion tool) Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, csabella, jayvdb, jstasiak, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-07-12 10:29 by jstasiak, last changed 2017-11-15 14:17 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 60 closed jstasiak, 2017-02-15 21:42
PR 3771 merged jstasiak, 2017-09-26 19:24
Messages (7)
msg270236 - (view) Author: Jakub Stasiak (jstasiak) * Date: 2016-07-12 10:29
Test file (test.py):

print(set(x for x in range(2),))

Python runs it nicely:

% python2 test.py 
set([0, 1])
% python3 test.py
{0, 1}

2to3 parser (on both Python 2.7.11 and 3.5.2) chokes on it though:

% /usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/bin/2to3 test.py 
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=u')', context=('', (1, 30))
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=u')', context=('', (1, 30))

% /usr/local/Cellar/python3/3.5.2/Frameworks/Python.framework/Versions/3.5/bin/2to3 test.py 
RefactoringTool: Skipping optional fixer: buffer
RefactoringTool: Skipping optional fixer: idioms
RefactoringTool: Skipping optional fixer: set_literal
RefactoringTool: Skipping optional fixer: ws_comma
RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=')', context=('', (1, 30))
RefactoringTool: No files need to be modified.
RefactoringTool: There was 1 error:
RefactoringTool: Can't parse test.py: ParseError: bad input: type=8, value=')', context=('', (1, 30))


For reference: https://github.com/smarkets/flake8-strict/issues/9 (project using lib2to3 parser)
msg295246 - (view) Author: Cheryl Sabella (csabella) * Date: 2017-06-06 11:10
Adding benjamin.peterson as he is listed as the expert for 2to3.
msg303058 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-26 18:11
set(x for x in range(2),) can be interpreted as set(x for x in (range(2),)).

Wouldn't be better to forbid such ambiguous syntax? The trailing comma in argument list is supported because it helps to add new arguments (or temporary comment out arguments).

    foo(x,
        y,
        #z,
       )

But set(x for x in range(2),) is not syntactically valid if add an argument after the comma. Parenthesis around a generator expression can be omitted only if it is the only argument in a function call. I think that it would be better to forbid a trailing comma in this case.
msg303500 - (view) Author: Jakub Stasiak (jstasiak) * Date: 2017-10-01 22:01
By "forbid" do you mean "forbid in Python" (as in change Python syntax)? I like the idea but that seems like a more serious change and 2to3 arguably needs to handle code targeting older Python versions anyway.
msg303744 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-10-05 07:10
New changeset af810b35b494ef1d255d4bf340b92a9dad446995 by Benjamin Peterson (Jakub Stasiak) in branch 'master':
closes bpo-27494: Fix 2to3 handling of trailing comma after a generator expression (#3771)
https://github.com/python/cpython/commit/af810b35b494ef1d255d4bf340b92a9dad446995
msg306146 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-13 07:39
Actually this syntax isn't allowed by the Python language specification. See issue32012 for fixing the Python parser.
msg306269 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-15 14:17
Since this syntax never was valid according to the Python language specification, and it causes SyntaxError in 3.7 (see issue32012), I think that this change should be reverted.
History
Date User Action Args
2017-11-15 14:17:32serhiy.storchakasetstatus: closed -> open
resolution: fixed ->
messages: + msg306269

stage: resolved ->
2017-11-13 07:39:22serhiy.storchakasetmessages: + msg306146
2017-10-05 07:10:11benjamin.petersonsetstatus: open -> closed
resolution: fixed
messages: + msg303744

stage: patch review -> resolved
2017-10-01 22:01:07jstasiaksetmessages: + msg303500
2017-09-26 19:24:12jstasiaksetkeywords: + patch
stage: patch review
pull_requests: + pull_request3758
2017-09-26 18:11:47serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg303058
2017-06-06 11:10:55csabellasetnosy: + csabella, benjamin.peterson
messages: + msg295246
2017-02-15 22:03:42jstasiaksetversions: + Python 3.6, Python 3.7
2017-02-15 21:42:37jstasiaksetpull_requests: + pull_request87
2016-07-12 10:34:44jayvdbsetnosy: + jayvdb
2016-07-12 10:29:13jstasiakcreate