Title: ast col_offset wrong for list comprehensions, generators and tuples
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: benjamin.peterson, brett.cannon, samuelcolvin, serhiy.storchaka, yselivanov
Priority: normal Keywords: patch

Created on 2017-08-20 19:03 by samuelcolvin, last changed 2018-11-27 07:42 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 10633 merged serhiy.storchaka, 2018-11-21 12:10
Messages (5)
msg300606 - (view) Author: Samuel Colvin (samuelcolvin) * Date: 2017-08-20 19:03
With Python 3.5 and 3.6 list comprehensions, generators and tuples have the col_offset for their ast nodes off by 1:

import ast
ast.parse('{a for a in range(3)}').body[0].value.col_offset
>> 0  # set comprehension correct

ast.parse('{a: 1 for a in range(3)}').body[0].value.col_offset
>> 0  # dict comprehension correct

ast.parse('[a for a in range(3)]').body[0].value.col_offset
>> 1  # list comprehension wrong!

ast.parse('(a for a in range(3))').body[0].value.col_offset
>> 1  # generator comprehension wrong!

ast.parse('[1, 2, 3]').body[0].value.col_offset
>> 0  # list correct

ast.parse('{1, 2, 3}').body[0].value.col_offset
>> 0  # set correct

ast.parse('{1: 1, 2: 2, 3: 3}').body[0].value.col_offset
>> 0  # dict correct

ast.parse('(1, 2, 3)').body[0].value.col_offset
>> 1  # tuple wrong!

I haven't tried 3.4, the issue could be there too.

There are some other related issues #16806 and #21295 but they don't seem quite the same.
msg330194 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-21 12:20
For list comprehensions and generator expressions this is definitely a bug. But tuples syntax technically does not include surrounded parentheses.

There is also a problem with generator expression passes as a single argument. Generator expression parentheses can be collapsed with function call parentheses: f(a for a in b).

PR 10633 makes the following changes:

* Fixes position for list comprehensions and generator expressions.

* If generator expression parentheses are be collapsed with function call parentheses, the position of the AST node for the generator expression points to the left parenthesis.

* For tuples surrounded with parentheses, the position of the AST node points to the left brace. For tuples without parentheses, it points to the position of the first tuple item.

I am not sure whether these changes should be backported to maintained versions.
msg330385 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-25 09:29
I am not sure what parts of this PR should be backported if either.
msg330470 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-11-26 21:54
Yeah, it's a tough call because it's one of those things others have probably worked around already, so backporting would break the work-arounds.

If you don't want to bother backporting, Serhiy, I think it would be fine to not do it.
msg330497 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-27 07:40
New changeset b619b097923155a7034c05c4018bf06af9f994d0 by Serhiy Storchaka in branch 'master':
bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
Date User Action Args
2018-11-27 07:42:56serhiy.storchakasetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.8, - Python 3.5, Python 3.6
2018-11-27 07:40:35serhiy.storchakasetmessages: + msg330497
2018-11-26 21:54:31brett.cannonsetmessages: + msg330470
2018-11-25 09:29:41serhiy.storchakasetmessages: + msg330385
2018-11-21 12:20:29serhiy.storchakasetnosy: + brett.cannon, benjamin.peterson, yselivanov
messages: + msg330194
2018-11-21 12:10:03serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request9880
2018-11-21 08:47:50serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
2017-08-20 19:03:53samuelcolvincreate