Incorrect offsets in new parser for f-string substitutions #86972

asottile · 2021-01-02T00:00:22Z

BPO	42806
Nosy	@gvanrossum, @asottile, @lysnikolaou, @pablogsal
PRs	bpo-42806: Fix ast locations of f-strings inside parentheses #24067 [3.9] bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067) #24069

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-03.01:32:51.560>
created_at = <Date 2021-01-02.00:00:22.370>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10']
title = 'Incorrect offsets in new parser for f-string substitutions'
updated_at = <Date 2021-01-03.01:33:09.167>
user = 'https://github.com/asottile'

bugs.python.org fields:

activity = <Date 2021-01-03.01:33:09.167>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-03.01:32:51.560>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-01-02.00:00:22.370>
creator = 'Anthony Sottile'
dependencies = []
files = []
hgrepos = []
issue_num = 42806
keywords = ['patch']
message_count = 8.0
messages = ['384208', '384209', '384210', '384211', '384229', '384246', '384249', '384250']
nosy_count = 4.0
nosy_names = ['gvanrossum', 'Anthony Sottile', 'lys.nikolaou', 'pablogsal']
pr_nums = ['24067', '24069']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue42806'
versions = ['Python 3.9', 'Python 3.10']

asottile · 2021-01-02T00:00:22Z

here's a sample file. this is distilled down from an example which broke a code formatter here: asottile/add-trailing-comma#106

def f():
    x = (
            'PERL_MM_OPT', (
                f'INSTALL-BASE={shlex.quote(venv)} '
                f'wat'
            ),
    )

A diff from astpretty (https://github.com/asottile/astpretty) between the old and new parser:

$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(.tox/py39/bin/python -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:58:09.108060968 -0800
+++ /dev/fd/62	2021-01-01 15:58:09.108060968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,

the old parser is correct here, and the new parser is wrong

notably it thinks that the call inside the f-string substitution is positioned at column 16 which is the position of the f-string start

this bug is also present in 3.10:

$ venv/bin/python3.10 --version
Python 3.10.0a3
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(venv/bin/python3.10 -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:59:54.000059968 -0800
+++ /dev/fd/62	2021-01-01 15:59:54.000059968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,

asottile · 2021-01-02T00:23:45Z

+peg parser individuals to nosy

gvanrossum · 2021-01-02T00:34:55Z

Did you check this with master?

asottile · 2021-01-02T00:43:46Z

I didn't initially, but now I have and it has the same bug in the current master: 3bf0532

asottile · 2021-01-02T16:53:04Z

by the way, here's a much smaller case which has a similar problem

x = (
    f" {test(t)}"
)

pablogsal · 2021-01-03T00:19:23Z

Seems that I also found this weird behaviour in the old parser:

import ast
code = """\
x = (
        'PERL_MM_OPT', (
            f'wat'
            f'INSTALL-BASE={shlex.quote(venv)} '
            f'wat'
        ),
)
"""
elem = ast.parse(code).body[0].value.elts[1].values[1].value
print(elem)
print(code.split("\n")[elem.lineno-1][elem.col_offset:elem.end_col_offset])

In Python3.8 this prints:

<_ast.Call object at 0x7f7484393a00>
'wat'

which is wrong as the code for that call is certainly not "wat". This happens in the oldparser in 3.9:

❯ ../3.9/python -Xoldparser lel.py
<ast.Call object at 0x7f78afbc1e10>
'wat'

And something wrong happens in the current master with the new parser:

❯ ../3.9/python lel.py
<ast.Call object at 0x7f504e9a1eb0>
STALL-BASE={shlex

But with PR24067:

<ast.Call object at 0x7fec78673fa0>
shlex.quote(venv)

pablogsal · 2021-01-03T01:12:11Z

New changeset bd2728b by Pablo Galindo in branch 'master':
bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067)
bd2728b

pablogsal · 2021-01-03T01:33:09Z

New changeset bfc413c by Pablo Galindo in branch '3.9':
[3.9] bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067) (GH-24069)
bfc413c

asottile mannequin added 3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Jan 2, 2021

pablogsal closed this as completed Jan 3, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorrect offsets in new parser for f-string substitutions #86972

Incorrect offsets in new parser for f-string substitutions #86972

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

gvanrossum commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

pablogsal commented Jan 3, 2021

pablogsal commented Jan 3, 2021

pablogsal commented Jan 3, 2021

Incorrect offsets in new parser for f-string substitutions #86972

Incorrect offsets in new parser for f-string substitutions #86972

Comments

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

gvanrossum commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

asottile mannequin commented Jan 2, 2021

pablogsal commented Jan 3, 2021

pablogsal commented Jan 3, 2021

pablogsal commented Jan 3, 2021