classification
Title: Expressions with format specifiers in f-strings give wrong code position in AST
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: arminius, asmeurer, blueyed, eric.smith, fhsxfhsx
Priority: normal Keywords: patch

Created on 2018-11-11 12:34 by arminius, last changed 2021-01-18 08:53 by blueyed.

Pull Requests
URL Status Linked Edit
PR 19398 open fhsxfhsx, 2020-04-06 18:20
Messages (3)
msg329672 - (view) Author: Arminius (arminius) Date: 2018-11-11 12:34
ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier.

Compare the trees of f'{a}' and f'{a:b}' :

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets,

Name(lineno=1, col_offset=3, id='a', ctx=Load())

and

Name(lineno=1, col_offset=1, id='a', ctx=Load())

.

Expected behavior: col_offset=3 for name "a" in both f-strings.

(In my specific use case, this breaks a semantic highlighting plugin for vim.)
msg353908 - (view) Author: Arminius (arminius) Date: 2019-10-04 08:53
There is another instance of incorrectly reported offsets in f-strings, in this case affecting multi-line strings.

Example:

(
    f'aaa{foo}bbb'
    f'ccc{bar}ddd'
    f'eee{baz}fff'
)

Relevant part of the syntax tree:

JoinedStr(
    lineno=2,
    col_offset=4,
    values=[
        Str(lineno=2, col_offset=4, s='aaa'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=10, id='foo', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='bbbccc'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=5, id='bar', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='dddcccaaaaaaaaaaaaaa'),
        FormattedValue(
            lineno=2,
            col_offset=4,
            value=Name(lineno=2, col_offset=5, id='baz', ctx=Load()),
            conversion=-1,
            format_spec=None,
        ),
        Str(lineno=2, col_offset=4, s='ddd'),
    ],
),

The reported code position for "foo" is correct (2,10), but not for "bar" (2,5) and "baz" (2,5).


Could you give an estimate as to whether this issue will be looked at any time soon?
msg376163 - (view) Author: Aaron Meurer (asmeurer) Date: 2020-08-31 19:00
The same thing occurs with specifiers like {a!r}.
History
Date User Action Args
2021-01-18 08:53:33blueyedsetnosy: + blueyed
2020-08-31 19:00:25asmeurersetnosy: + asmeurer
messages: + msg376163
2020-04-06 18:20:31fhsxfhsxsetkeywords: + patch
nosy: + fhsxfhsx

pull_requests: + pull_request18760
stage: patch review
2019-10-04 08:53:26arminiussetmessages: + msg353908
2018-11-11 13:54:10eric.smithsetassignee: eric.smith
2018-11-11 13:51:18xtreaksetnosy: + eric.smith
2018-11-11 12:34:00arminiuscreate