classification
Title: Output of syntax error in f-string contains wrong filename
Type: behavior Stage: resolved
Components: Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.smith Nosy List: BTaskaya, Erik Cederstrand, eric.smith, lys.nikolaou, pablogsal
Priority: normal Keywords: patch

Created on 2019-12-04 08:27 by Erik Cederstrand, last changed 2020-05-26 06:51 by lys.nikolaou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20399 merged lys.nikolaou, 2020-05-25 21:45
PR 20404 merged lys.nikolaou, 2020-05-26 00:41
Messages (9)
msg357777 - (view) Author: Erik Cederstrand (Erik Cederstrand) Date: 2019-12-04 08:27
When I have a normal syntax error in a file, Python reports the filename in the exception output:

$ cat syntax_error.py 
0x=5

$ python3.8 syntax_error.py 
  File "syntax_error.py", line 1
    0x=5
     ^
SyntaxError: invalid hexadecimal literal


But if the syntax error is inside an f-string, Python reports 'File "<fstring>"' instead of the actual filename in the exception output.

$ cat syntax_error_in_fstring.py 
f'This is a syntax error: {0x=5}'

$ python3.8 syntax_error_in_fstring.py 
  File "<fstring>", line 1
SyntaxError: invalid hexadecimal literal
msg357780 - (view) Author: Erik Cederstrand (Erik Cederstrand) Date: 2019-12-04 08:39
Additionally, the output in the 2nd example does not contain the helpful text printing the context and location of the code containing the syntax error.
msg357801 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2019-12-04 12:08
This is actually specified behavior in PEP 498
 
https://www.python.org/dev/peps/pep-0498/#expression-evaluation

> Expressions are parsed with the equivalent of ast.parse('(' + expression + ')', '<fstring>', 'eval')
msg357804 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-04 13:18
Although Batuhan is correct maybe we can consider trying to improve the debugging experience off-strings due to the fact that these errors in a big project may be difficult to find. Although you have always the back-trace on the other side.
msg357805 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-12-04 13:23
I agree that although the PEP says that's how it behaves, that shouldn't be prescriptive of the error message. Clearly we can be more helpful here.

I have a large, elaborate re-write of the error generating code that I've been working on, on and off for over a year, which would address this problem. I'm not sure when or if I'll finish it. It touches a lot of things, and my branch won't merge cleanly any more.

Additionally, the idea of moving f-strings into the Python grammar would also fix this problem. That might be a better way forward, although it has a number of downsides, too.
msg357806 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-04 15:06
> Additionally, the idea of moving f-strings into the Python grammar would also fix this problem. That might be a better way forward, although it has a number of downsides, too.

For considerin this path we would need to wait to see first the future of the new experiment with the PEG parser to avoid potentially throwing away all the work in the parser and the tokenizer. Also, this approach will likely entail much more changes than reworking the error reporting code.
msg357821 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2019-12-04 20:05
Yes, moving f-strings to the grammar would be a huge change, and not just for CPython.

I discussed it at the last PyCon with the authors of various editors (for syntax highlighting) and other tools that parse python code. No one was hugely opposed to it, and I think even one person was very excited about it. But it's not a step to be taken lightly. I don't think I've discussed it with any other Python implementors outside of CPython.

My biggest concern is that it makes naive string recognition fail. For example, this would become a valid f-string:
f'{fn('some string')}'
while
'{fn('some string')}'
is not a valid string.

I now I think I've completely derailed this bug report. I think the action item here is for me to finish up my better error reporting.
msg369933 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-26 00:32
New changeset f7b1e461567e5e3fa3ba46f589d9edc1b45b2dd0 by Lysandros Nikolaou in branch 'master':
bpo-38964: Print correct filename on a SyntaxError in an fstring (GH-20399)
https://github.com/python/cpython/commit/f7b1e461567e5e3fa3ba46f589d9edc1b45b2dd0
msg369936 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-26 01:24
New changeset 791a46ed58f74d673cf2c0d81deec57155bf7583 by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-38964: Print correct filename on a SyntaxError in an fstring (GH-20399) (GH-20404)
https://github.com/python/cpython/commit/791a46ed58f74d673cf2c0d81deec57155bf7583
History
Date User Action Args
2020-05-26 06:51:28lys.nikolaousetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-05-26 01:24:39pablogsalsetmessages: + msg369936
2020-05-26 00:41:23lys.nikolaousetpull_requests: + pull_request19666
2020-05-26 00:32:25pablogsalsetmessages: + msg369933
2020-05-25 21:45:56lys.nikolaousetversions: + Python 3.10, - Python 3.8
2020-05-25 21:45:18lys.nikolaousetkeywords: + patch
nosy: + lys.nikolaou

pull_requests: + pull_request19662
stage: patch review
2019-12-04 20:05:25eric.smithsetmessages: + msg357821
2019-12-04 15:06:44pablogsalsetmessages: + msg357806
2019-12-04 13:23:34eric.smithsetassignee: eric.smith
stage: resolved -> (no value)
2019-12-04 13:23:15eric.smithsetmessages: + msg357805
2019-12-04 13:18:30pablogsalsetstatus: closed -> open
resolution: fixed -> (no value)
2019-12-04 13:18:18pablogsalsetstatus: open -> closed

nosy: + pablogsal
messages: + msg357804

resolution: fixed
stage: resolved
2019-12-04 12:08:14BTaskayasetnosy: + BTaskaya
messages: + msg357801
2019-12-04 08:46:12xtreaksetnosy: + eric.smith
2019-12-04 08:39:59Erik Cederstrandsetmessages: + msg357780
2019-12-04 08:27:53Erik Cederstrandcreate