-
-
Notifications
You must be signed in to change notification settings - Fork 29.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect offsets in new parser for f-string substitutions #86972
Comments
here's a sample file. this is distilled down from an example which broke a code formatter here: asottile/add-trailing-comma#106 def f():
x = (
'PERL_MM_OPT', (
f'INSTALL-BASE={shlex.quote(venv)} '
f'wat'
),
) A diff from astpretty (https://github.com/asottile/astpretty) between the old and new parser: $ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(.tox/py39/bin/python -m astpretty perl.py)
--- /dev/fd/63 2021-01-01 15:58:09.108060968 -0800
+++ /dev/fd/62 2021-01-01 15:58:09.108060968 -0800
@@ -35,19 +35,19 @@
end_col_offset=22,
value=Call(
lineno=4,
- col_offset=32,
+ col_offset=16,
end_lineno=4,
- end_col_offset=49,
+ end_col_offset=33,
func=Attribute(
lineno=4,
- col_offset=32,
+ col_offset=16,
end_lineno=4,
- end_col_offset=43,
- value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+ end_col_offset=27,
+ value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
attr='quote',
ctx=Load(),
),
- args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+ args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
keywords=[],
),
conversion=-1, the old parser is correct here, and the new parser is wrong notably it thinks that the call inside the f-string substitution is positioned at column 16 which is the position of the f-string start this bug is also present in 3.10: $ venv/bin/python3.10 --version
Python 3.10.0a3
$ diff -u <(.tox/py39/bin/python -X oldparser -m astpretty perl.py) <(venv/bin/python3.10 -m astpretty perl.py)
--- /dev/fd/63 2021-01-01 15:59:54.000059968 -0800
+++ /dev/fd/62 2021-01-01 15:59:54.000059968 -0800
@@ -35,19 +35,19 @@
end_col_offset=22,
value=Call(
lineno=4,
- col_offset=32,
+ col_offset=16,
end_lineno=4,
- end_col_offset=49,
+ end_col_offset=33,
func=Attribute(
lineno=4,
- col_offset=32,
+ col_offset=16,
end_lineno=4,
- end_col_offset=43,
- value=Name(lineno=4, col_offset=32, end_lineno=4, end_col_offset=37, id='shlex', ctx=Load()),
+ end_col_offset=27,
+ value=Name(lineno=4, col_offset=16, end_lineno=4, end_col_offset=21, id='shlex', ctx=Load()),
attr='quote',
ctx=Load(),
),
- args=[Name(lineno=4, col_offset=44, end_lineno=4, end_col_offset=48, id='venv', ctx=Load())],
+ args=[Name(lineno=4, col_offset=28, end_lineno=4, end_col_offset=32, id='venv', ctx=Load())],
keywords=[],
),
conversion=-1, |
+peg parser individuals to nosy |
Did you check this with master? |
I didn't initially, but now I have and it has the same bug in the current master: 3bf0532 |
by the way, here's a much smaller case which has a similar problem
|
Seems that I also found this weird behaviour in the old parser: import ast
code = """\
x = (
'PERL_MM_OPT', (
f'wat'
f'INSTALL-BASE={shlex.quote(venv)} '
f'wat'
),
)
"""
elem = ast.parse(code).body[0].value.elts[1].values[1].value
print(elem)
print(code.split("\n")[elem.lineno-1][elem.col_offset:elem.end_col_offset]) In Python3.8 this prints: <_ast.Call object at 0x7f7484393a00> which is wrong as the code for that call is certainly not "wat". This happens in the oldparser in 3.9: ❯ ../3.9/python -Xoldparser lel.py And something wrong happens in the current master with the new parser: ❯ ../3.9/python lel.py But with PR24067: <ast.Call object at 0x7fec78673fa0> |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: