classification
Title: Specialise syntax error of **dict in f-string field
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: JNCressey, epaine, eric.smith, pablogsal, terry.reedy
Priority: normal Keywords: patch

Created on 2020-06-21 12:32 by JNCressey, last changed 2021-03-24 19:34 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 25006 merged pablogsal, 2021-03-24 00:32
Messages (7)
msg371995 - (view) Author: JNCressey (JNCressey) Date: 2020-06-21 12:32
Compare f"{*my_tuple}" with f"{**my_dict}".
Both are syntax errors since you can't use unpacking here, but the bug is as follows:

- For the tuple, IDLE highlights the asterisk and has the helpful message 'SyntaxError: can't use starred expression here',

- But for the dictionary, the first few characters of your code are highlighted, regardless of where the syntax error is located, and the message only says 'SyntaxError: invalid syntax'.

Bug occurs in both 3.8.3 and 3.7.7, I haven't tested it in 3.6 nor in-development versions.
msg371999 - (view) Author: E. Paine (epaine) * Date: 2020-06-21 15:29
This isn't an IDLE issue and is instead due to the core interpreter. This isn't a bug, there just isn't a more specialised error message for the dictionary unpacking failure.

Changing the error message to something more helpful would be an enhancement and therefore would not be back-ported to Python 3.9 and before.
msg372007 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-06-21 17:23
IDLE highlights the location of a SyntaxError according to its .lineno and .offset attributes.  For expressions within braces within f strings, the reported offset is respect to the expression augmented with surrounding ()s added, not the actual physical line.  From my perspective as IDLE maintainer, I consider this undocumented 'feature' a design bug.

Pablo, I know you did not do this, but you helped design and understand the new compiler.  If we cannot change .offset to report the real offset, can we add a new attribute, .line_offset for 3.9 and 3.10?  Does the new compiler know where within the line the expression starts?  Are there any other cases where .offset is not the real offset with respect to the actual code?  Or rather, are there other cases where the actual line is replaced with a constructed line?  Is so, are confusing fake ()s always added, that make the replacement *not* a substring of the original?  Are there any plans to add new cases like this?

In the absence of compile providing the line offset, every IDE that marks errors in the actual line has to separately code a workaround.  However, a change in 3.9 makes this harder.  In 3.8, errors within fstring expressions were tagged with .filename '<fstring>'.  In 3.9, this is attribute is ''.


The REPL in 3.7 (very soon security fixes only) and 3.8:
>>> f'{*a}'
  File "<stdin>", line 1
SyntaxError: can't use starred expression here
>>> f'{**a}'
  File "<fstring>", line 1
    (**a)
     ^
SyntaxError: invalid syntax
>>> f'{a a}'
  File "<fstring>", line 1
    (a a)
       ^
SyntaxError: invalid syntax


The REPL in 3.9/3.10 with the new parser is more consistent.
>>> f'{*a}'
  File "<stdin>", line 1
    (*a)                  # Does not ^ belong under *?  See below.
       ^
SyntaxError: invalid syntax
>>> f"{'abc' + *a}"
  File "<stdin>", line 1
    ('abc' + *a)          # This was the same in 3.8.
             ^
SyntaxError: invalid syntax
msg389366 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-03-23 02:11
Cressey noted both a compile and IDLE Shell issue.  They must be handled separately by different people in separate bpo issues.

The first is about the helpful versus less helpful SyntaxError messages in the following (master compiled today).

>>> f'{*x}'
  File "<stdin>", line 1
    (*x)
     ^
SyntaxError: f-string: can't use starred expression here
>>> f'{**x}'
  File "<stdin>", line 1
    (**x)
     ^
SyntaxError: f-string: invalid syntax

The request is make the 2nd message the same or like the 1st.  Seems like it should be possible.  I am limiting this bpo issue to this request and opened #43600 for fixing IDLE's highlight location.

Pablo, I leave it to you to handle the message enhancement request.  Do you know if there is any other situation like this in which the compiled text is not the input code?
msg389375 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-03-23 08:04
The parens are added in fstring_compile_expr at https://github.com/python/cpython/blob/9feae41c4f04ca27fd2c865807a5caeb50bf4fc4/Parser/string_parser.c#L391

I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required. If it's just a whitespace issue, maybe skipping leading whitespace in fstring_compile_expr is a better idea? One of the reasons I originally didn't want to skip leading whitespace is because I didn't want to add yet another place where we'd have to maintain a "what qualifies as whitespace to the compiler?" decision.
msg389421 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-24 00:35
> I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required.

IIRC, this forces the expression inside to be parsed as an expression. This helps the quite a lot the parser. For instance, detecting a bare **x without grouping is more difficult, even with the old parser. With parentheses it must be an expression so is easier to disambiguate.
msg389486 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-03-24 19:34
New changeset 8efad61963809d239cac986e3f3bc4cb505ab8fe by Pablo Galindo in branch 'master':
bpo-41064: Improve syntax error for invalid usage of '**' in f-strings (GH-25006)
https://github.com/python/cpython/commit/8efad61963809d239cac986e3f3bc4cb505ab8fe
History
Date User Action Args
2021-03-24 19:34:42pablogsalsetmessages: + msg389486
2021-03-24 19:34:39pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-03-24 00:35:25pablogsalsetmessages: + msg389421
2021-03-24 00:32:52pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request23764
2021-03-23 08:04:03eric.smithsetmessages: + msg389375
2021-03-23 02:11:13terry.reedysetassignee: terry.reedy ->
messages: + msg389366
components: - IDLE
title: f-string SyntaxError gives offset within fake line, other issues -> Specialise syntax error of **dict in f-string field
2020-06-21 18:08:12eric.smithsetnosy: + eric.smith
2020-06-21 17:24:30terry.reedysetnosy: + epaine
2020-06-21 17:23:51terry.reedysetversions: + Python 3.8, Python 3.9
nosy: + pablogsal, - epaine
title: Specialise syntax error of ** unpacking dict -> f-string SyntaxError gives offset within fake line, other issues
messages: + msg372007

components: + IDLE
type: enhancement -> behavior
2020-06-21 15:29:00epainesettype: behavior -> enhancement
title: IDLE highlights wrong place when code has syntax error of ** unpacking dict in f-string -> Specialise syntax error of ** unpacking dict
components: + Interpreter Core, - IDLE
versions: + Python 3.10, - Python 3.7, Python 3.8
nosy: + epaine

messages: + msg371999
2020-06-21 12:57:09SilentGhostsettype: compile error -> behavior
2020-06-21 12:32:38JNCresseycreate