Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specialise syntax error of **dict in f-string field #85236

Closed
JNCressey mannequin opened this issue Jun 21, 2020 · 7 comments
Closed

Specialise syntax error of **dict in f-string field #85236

JNCressey mannequin opened this issue Jun 21, 2020 · 7 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@JNCressey
Copy link
Mannequin

JNCressey mannequin commented Jun 21, 2020

BPO 41064
Nosy @terryjreedy, @ericvsmith, @pablogsal, @E-Paine, @JNCressey
PRs
  • bpo-41064: Improve syntax error for invalid usage of '**' in f-strings #25006
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2021-03-24.19:34:39.459>
    created_at = <Date 2020-06-21.12:32:38.555>
    labels = ['interpreter-core', 'type-bug', '3.8', '3.9', '3.10']
    title = 'Specialise syntax error of **dict in f-string field'
    updated_at = <Date 2021-03-24.19:34:42.616>
    user = 'https://github.com/JNCressey'

    bugs.python.org fields:

    activity = <Date 2021-03-24.19:34:42.616>
    actor = 'pablogsal'
    assignee = 'none'
    closed = True
    closed_date = <Date 2021-03-24.19:34:39.459>
    closer = 'pablogsal'
    components = ['Interpreter Core']
    creation = <Date 2020-06-21.12:32:38.555>
    creator = 'JNCressey'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 41064
    keywords = ['patch']
    message_count = 7.0
    messages = ['371995', '371999', '372007', '389366', '389375', '389421', '389486']
    nosy_count = 5.0
    nosy_names = ['terry.reedy', 'eric.smith', 'pablogsal', 'epaine', 'JNCressey']
    pr_nums = ['25006']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue41064'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @JNCressey
    Copy link
    Mannequin Author

    JNCressey mannequin commented Jun 21, 2020

    Compare f"{*my_tuple}" with f"{**my_dict}".
    Both are syntax errors since you can't use unpacking here, but the bug is as follows:

    • For the tuple, IDLE highlights the asterisk and has the helpful message 'SyntaxError: can't use starred expression here',

    • But for the dictionary, the first few characters of your code are highlighted, regardless of where the syntax error is located, and the message only says 'SyntaxError: invalid syntax'.

    Bug occurs in both 3.8.3 and 3.7.7, I haven't tested it in 3.6 nor in-development versions.

    @JNCressey JNCressey mannequin added build The build process and cross-build 3.7 (EOL) end of life 3.8 only security fixes topic-IDLE labels Jun 21, 2020
    @JNCressey JNCressey mannequin assigned terryjreedy Jun 21, 2020
    @SilentGhost SilentGhost mannequin added type-bug An unexpected behavior, bug, or error and removed build The build process and cross-build labels Jun 21, 2020
    @E-Paine
    Copy link
    Mannequin

    E-Paine mannequin commented Jun 21, 2020

    This isn't an IDLE issue and is instead due to the core interpreter. This isn't a bug, there just isn't a more specialised error message for the dictionary unpacking failure.

    Changing the error message to something more helpful would be an enhancement and therefore would not be back-ported to Python 3.9 and before.

    @E-Paine E-Paine mannequin added 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) and removed 3.7 (EOL) end of life 3.8 only security fixes topic-IDLE labels Jun 21, 2020
    @E-Paine E-Paine mannequin changed the title IDLE highlights wrong place when code has syntax error of ** unpacking dict in f-string Specialise syntax error of ** unpacking dict Jun 21, 2020
    @E-Paine E-Paine mannequin added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error topic-IDLE labels Jun 21, 2020
    @E-Paine E-Paine mannequin changed the title IDLE highlights wrong place when code has syntax error of ** unpacking dict in f-string Specialise syntax error of ** unpacking dict Jun 21, 2020
    @E-Paine E-Paine mannequin added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Jun 21, 2020
    @terryjreedy
    Copy link
    Member

    IDLE highlights the location of a SyntaxError according to its .lineno and .offset attributes. For expressions within braces within f strings, the reported offset is respect to the expression augmented with surrounding ()s added, not the actual physical line. From my perspective as IDLE maintainer, I consider this undocumented 'feature' a design bug.

    Pablo, I know you did not do this, but you helped design and understand the new compiler. If we cannot change .offset to report the real offset, can we add a new attribute, .line_offset for 3.9 and 3.10? Does the new compiler know where within the line the expression starts? Are there any other cases where .offset is not the real offset with respect to the actual code? Or rather, are there other cases where the actual line is replaced with a constructed line? Is so, are confusing fake ()s always added, that make the replacement *not* a substring of the original? Are there any plans to add new cases like this?

    In the absence of compile providing the line offset, every IDE that marks errors in the actual line has to separately code a workaround. However, a change in 3.9 makes this harder. In 3.8, errors within fstring expressions were tagged with .filename '<fstring>'. In 3.9, this is attribute is ''.

    The REPL in 3.7 (very soon security fixes only) and 3.8:
    >>> f'{*a}'
      File "<stdin>", line 1
    SyntaxError: can't use starred expression here
    >>> f'{**a}'
      File "<fstring>", line 1
        (**a)
         ^
    SyntaxError: invalid syntax
    >>> f'{a a}'
      File "<fstring>", line 1
        (a a)
           ^
    SyntaxError: invalid syntax
    
    
    The REPL in 3.9/3.10 with the new parser is more consistent.
    >>> f'{*a}'
      File "<stdin>", line 1
        (*a)                  # Does not ^ belong under *?  See below.
           ^
    SyntaxError: invalid syntax
    >>> f"{'abc' + *a}"
      File "<stdin>", line 1
        ('abc' + *a)          # This was the same in 3.8.
                 ^
    SyntaxError: invalid syntax

    @terryjreedy terryjreedy added topic-IDLE 3.8 only security fixes 3.9 only security fixes labels Jun 21, 2020
    @terryjreedy terryjreedy changed the title Specialise syntax error of ** unpacking dict f-string SyntaxError gives offset within fake line, other issues Jun 21, 2020
    @terryjreedy terryjreedy added topic-IDLE 3.8 only security fixes 3.9 only security fixes and removed type-feature A feature request or enhancement labels Jun 21, 2020
    @terryjreedy terryjreedy added the type-bug An unexpected behavior, bug, or error label Jun 21, 2020
    @terryjreedy terryjreedy changed the title Specialise syntax error of ** unpacking dict f-string SyntaxError gives offset within fake line, other issues Jun 21, 2020
    @terryjreedy terryjreedy added type-bug An unexpected behavior, bug, or error and removed type-feature A feature request or enhancement labels Jun 21, 2020
    @terryjreedy
    Copy link
    Member

    Cressey noted both a compile and IDLE Shell issue. They must be handled separately by different people in separate bpo issues.

    The first is about the helpful versus less helpful SyntaxError messages in the following (master compiled today).

    >>> f'{*x}'
      File "<stdin>", line 1
        (*x)
         ^
    SyntaxError: f-string: can't use starred expression here
    >>> f'{**x}'
      File "<stdin>", line 1
        (**x)
         ^
    SyntaxError: f-string: invalid syntax

    The request is make the 2nd message the same or like the 1st. Seems like it should be possible. I am limiting this bpo issue to this request and opened bpo-43600 for fixing IDLE's highlight location.

    Pablo, I leave it to you to handle the message enhancement request. Do you know if there is any other situation like this in which the compiled text is not the input code?

    @terryjreedy terryjreedy changed the title f-string SyntaxError gives offset within fake line, other issues Specialise syntax error of **dict in f-string field Mar 23, 2021
    @terryjreedy terryjreedy removed their assignment Mar 23, 2021
    @terryjreedy terryjreedy changed the title f-string SyntaxError gives offset within fake line, other issues Specialise syntax error of **dict in f-string field Mar 23, 2021
    @terryjreedy terryjreedy removed their assignment Mar 23, 2021
    @ericvsmith
    Copy link
    Member

    The parens are added in fstring_compile_expr at

    // The parentheses are needed in order to allow for leading whitespace within

    I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required. If it's just a whitespace issue, maybe skipping leading whitespace in fstring_compile_expr is a better idea? One of the reasons I originally didn't want to skip leading whitespace is because I didn't want to add yet another place where we'd have to maintain a "what qualifies as whitespace to the compiler?" decision.

    @pablogsal
    Copy link
    Member

    I don't recall if this is really only a "skip leading whitespace" problem, or if there's some other reason they're required.

    IIRC, this forces the expression inside to be parsed as an expression. This helps the quite a lot the parser. For instance, detecting a bare **x without grouping is more difficult, even with the old parser. With parentheses it must be an expression so is easier to disambiguate.

    @pablogsal
    Copy link
    Member

    New changeset 8efad61 by Pablo Galindo in branch 'master':
    bpo-41064: Improve syntax error for invalid usage of '**' in f-strings (GH-25006)
    8efad61

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants