classification
Title: Error message differs when an expression is in an fstring
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, eric.smith, gvanrossum, lys.nikolaou, pablogsal
Priority: normal Keywords: patch

Created on 2020-04-12 21:59 by lys.nikolaou, last changed 2020-04-15 18:24 by gvanrossum. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19521 merged lys.nikolaou, 2020-04-14 18:50
Messages (3)
msg366272 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-12 21:59
There are cases, where the error message differs, when an expression is being parsed inside an fstring. For example:

>>> f'{x+}'
  File "<fstring>", line 1
    (x+)
       ^
SyntaxError: unexpected EOF while parsing
>>> (x+)
  File "<stdin>", line 1
    (x+)
       ^
SyntaxError: invalid syntax

Or with lambda definitions:

>>> f'{lambda x:x}'
  File "<fstring>", line 1
    (lambda x)
             ^
SyntaxError: unexpected EOF while parsing
>>> (lambda x)
  File "<stdin>", line 1
    (lambda x)
             ^
SyntaxError: invalid syntax
msg366273 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-12 22:41
It seems that this is actually a bit bigger than this and it is not specific to f-strings. 

The error message *always* changes to `unexpected EOF while parsing` if there is an error with the last character of the input and no newline follows. For example, as made clear to me by Guido, there are even differences in error messages between exec'ing and eval'ing something:

>>> exec('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: invalid syntax
>>> eval('x+')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+
     ^
SyntaxError: unexpected EOF while parsing

That's because the tokenizer adds an implicit newline to the input string, before tokenizing it, when the input comes from an exec call. (see https://github.com/python/cpython/blob/14d5331eb5e6c38be12bad421bd59ad0fac9e448/Parser/tokenizer.c#L648)

And that's not limited to a character missing, as suggested by the error message. Even when the last character itself generates a SyntaxError, the error message remains "unexpcted EOF while parsing":

>>> x+@
  File "<stdin>", line 1
    x+@
      ^
SyntaxError: invalid syntax
>>> eval('x+@')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    x+@
      ^
SyntaxError: unexpected EOF while parsing

Thus, a very simple fix to the specific fstring problem of this issue would be to add a newline to the string that gets parsed to the parser in fstring_compile_expr in ast.c, but I guess it'd be better to fix the tokenizer itself, if this is considered a bug.
msg366535 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-15 18:22
New changeset 9a4b38f66b3e674db94e07980e1cacb39e388c73 by Lysandros Nikolaou in branch 'master':
bpo-40267: Fix message when last input character produces a SyntaxError (GH-19521)
https://github.com/python/cpython/commit/9a4b38f66b3e674db94e07980e1cacb39e388c73
History
Date User Action Args
2020-04-15 18:24:17gvanrossumsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-04-15 18:22:17gvanrossumsetmessages: + msg366535
2020-04-14 18:50:22lys.nikolaousetkeywords: + patch
stage: patch review
pull_requests: + pull_request18871
2020-04-13 10:09:14BTaskayasetnosy: + BTaskaya
2020-04-13 09:01:32xtreaksetnosy: + eric.smith
2020-04-13 08:50:05lys.nikolaousettype: behavior
2020-04-12 22:41:03lys.nikolaousetmessages: + msg366273
2020-04-12 21:59:24lys.nikolaoucreate