With Python 3.5 and 3.6 list comprehensions, generators and tuples have the col_offset for their ast nodes off by 1:

import ast
ast.parse('{a for a in range(3)}').body[0].value.col_offset
>> 0  # set comprehension correct

ast.parse('{a: 1 for a in range(3)}').body[0].value.col_offset
>> 0  # dict comprehension correct

ast.parse('[a for a in range(3)]').body[0].value.col_offset
>> 1  # list comprehension wrong!

ast.parse('(a for a in range(3))').body[0].value.col_offset
>> 1  # generator comprehension wrong!

ast.parse('[1, 2, 3]').body[0].value.col_offset
>> 0  # list correct

ast.parse('{1, 2, 3}').body[0].value.col_offset
>> 0  # set correct

ast.parse('{1: 1, 2: 2, 3: 3}').body[0].value.col_offset
>> 0  # dict correct

ast.parse('(1, 2, 3)').body[0].value.col_offset
>> 1  # tuple wrong!

I haven't tried 3.4, the issue could be there too.

There are some other related issues bpo-16806 and bpo-21295 but they don't seem quite the same.

For list comprehensions and generator expressions this is definitely a bug. But tuples syntax technically does not include surrounded parentheses.

There is also a problem with generator expression passes as a single argument. Generator expression parentheses can be collapsed with function call parentheses: f(a for a in b).

PR 10633 makes the following changes:

• Fixes position for list comprehensions and generator expressions.

• If generator expression parentheses are be collapsed with function call parentheses, the position of the AST node for the generator expression points to the left parenthesis.

• For tuples surrounded with parentheses, the position of the AST node points to the left brace. For tuples without parentheses, it points to the position of the first tuple item.

I am not sure whether these changes should be backported to maintained versions.

I am not sure what parts of this PR should be backported if either.

Yeah, it's a tough call because it's one of those things others have probably worked around already, so backporting would break the work-arounds.

If you don't want to bother backporting, Serhiy, I think it would be fine to not do it.

New changeset b619b09 by Serhiy Storchaka in branch 'master':
bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
b619b09