This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Incorrect offsets in new parser for f-string substitutions
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, gvanrossum, lys.nikolaou, pablogsal
Priority: normal Keywords: patch

Created on 2021-01-02 00:00 by Anthony Sottile, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24067 merged pablogsal, 2021-01-03 00:02
PR 24069 merged pablogsal, 2021-01-03 01:13
Messages (8)
msg384208 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-01-02 00:00
here's a sample file.  this is distilled down from an example which broke a code formatter here: https://github.com/asottile/add-trailing-comma/issues/106

def f():
    x = (
            'PERL_MM_OPT', (
                f'INSTALL-BASE={shlex.quote(venv)} '
                f'wat'
            ),
    )


A diff from astpretty (https://github.com/asottile/astpretty) between the old and new parser:

```console
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(.tox/py39/bin/python -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:58:09.108060968 -0800
+++ /dev/fd/62	2021-01-01 15:58:09.108060968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,
```

the old parser is correct here, and the new parser is wrong

notably it thinks that the call inside the f-string substitution is positioned at column 16 which is the position of the f-string start

this bug is also present in 3.10:

```console
$ venv/bin/python3.10 --version
Python 3.10.0a3
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(venv/bin/python3.10 -m astpretty perl.py)
--- /dev/fd/63	2021-01-01 15:59:54.000059968 -0800
+++ /dev/fd/62	2021-01-01 15:59:54.000059968 -0800
@@ -35,19 +35,19 @@
                                         end_col_offset=22,
                                         value=Call(
                                             lineno=4,
-                                            col_offset=32,
+                                            col_offset=16,
                                             end_lineno=4,
-                                            end_col_offset=49,
+                                            end_col_offset=33,
                                             func=Attribute(
                                                 lineno=4,
-                                                col_offset=32,
+                                                col_offset=16,
                                                 end_lineno=4,
-                                                end_col_offset=43,
-                                                value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+                                                end_col_offset=27,
+                                                value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
                                                 attr='quote',
                                                 ctx=Load(),
                                             ),
-                                            args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+                                            args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
                                             keywords=[],
                                         ),
                                         conversion=-1,
```
msg384209 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-01-02 00:23
+peg parser individuals to nosy
msg384210 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-01-02 00:34
Did you check this with master?
msg384211 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-01-02 00:43
I didn't initially, but now I have and it has the same bug in the current master: 3bf05327c2b25d42b92795d9d280288c22a0963d
msg384229 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2021-01-02 16:53
by the way, here's a much smaller case which has a similar problem

```
x = (
    f" {test(t)}"
)
```
msg384246 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-01-03 00:19
Seems that I also found this weird behaviour in the old parser:

import ast
code = """\
x = (
        'PERL_MM_OPT', (
            f'wat'
            f'INSTALL-BASE={shlex.quote(venv)} '
            f'wat'
        ),
)
"""
elem = ast.parse(code).body[0].value.elts[1].values[1].value
print(elem)
print(code.split("\n")[elem.lineno-1][elem.col_offset:elem.end_col_offset])

In Python3.8 this prints:

<_ast.Call object at 0x7f7484393a00>
'wat'

which is wrong as the code for that call is certainly not "wat". This happens in the oldparser in 3.9:

❯ ../3.9/python -Xoldparser lel.py
<ast.Call object at 0x7f78afbc1e10>
'wat'

And something wrong happens in the current master with the new parser:

❯ ../3.9/python lel.py
<ast.Call object at 0x7f504e9a1eb0>
STALL-BASE={shlex

But with PR24067:

<ast.Call object at 0x7fec78673fa0>
shlex.quote(venv)
msg384249 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-01-03 01:12
New changeset bd2728b1e8a99ba8f8c2d481f88aeb99b8b8360f by Pablo Galindo in branch 'master':
bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067)
https://github.com/python/cpython/commit/bd2728b1e8a99ba8f8c2d481f88aeb99b8b8360f
msg384250 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-01-03 01:33
New changeset bfc413ce4fa37ccb889757388102c7755e057bf5 by Pablo Galindo in branch '3.9':
[3.9] bpo-42806: Fix ast locations of f-strings inside parentheses (GH-24067) (GH-24069)
https://github.com/python/cpython/commit/bfc413ce4fa37ccb889757388102c7755e057bf5
History
Date User Action Args
2022-04-11 14:59:39adminsetgithub: 86972
2021-01-03 01:33:09pablogsalsetmessages: + msg384250
2021-01-03 01:32:51pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-01-03 01:13:04pablogsalsetpull_requests: + pull_request22902
2021-01-03 01:12:11pablogsalsetmessages: + msg384249
2021-01-03 00:19:23pablogsalsetmessages: + msg384246
2021-01-03 00:02:11pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request22900
2021-01-02 16:53:03Anthony Sottilesetmessages: + msg384229
2021-01-02 00:43:45Anthony Sottilesetmessages: + msg384211
2021-01-02 00:34:54gvanrossumsetmessages: + msg384210
2021-01-02 00:23:44Anthony Sottilesetnosy: + gvanrossum, lys.nikolaou, pablogsal
messages: + msg384209
2021-01-02 00:00:22Anthony Sottilecreate