Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorrect exception highlighting for fstring format #89048

Closed
pablogsal opened this issue Aug 11, 2021 · 8 comments
Closed

Incorrect exception highlighting for fstring format #89048

pablogsal opened this issue Aug 11, 2021 · 8 comments
Labels
3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error

Comments

@pablogsal
Copy link
Member

BPO 44885
Nosy @gvanrossum, @ammaraskar, @lysnikolaou, @pablogsal, @isidentical
PRs
  • bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions #27729
  • [3.10] bpo-44885: Correct the ast locations of f-strings with format … #27743
  • [3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) #27744
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2021-08-12.16:42:19.299>
created_at = <Date 2021-08-11.07:59:02.051>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'Incorrect exception highlighting for fstring format'
updated_at = <Date 2021-08-12.17:46:49.800>
user = 'https://github.com/pablogsal'

    bugs.python.org fields:

    activity = <Date 2021-08-12.17:46:49.800>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-12.16:42:19.299>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-08-11.07:59:02.051>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 44885
keywords = ['patch']
message_count = 8.0
messages = ['399372', '399392', '399397', '399399', '399400', '399469', '399471', '399475']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ammar2', 'lys.nikolaou', 'pablogsal', 'BTaskaya']
pr_nums = ['27729', '27743', '27744']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44885'
versions = ['Python 3.11']
    @pablogsal
    Copy link
    Member Author

    Given this code:

    print(f"Here is that pesky {xxx/2:.3f} again")

    The traceback prints:

    Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/lel.py", line 1, in <module>
    print(f"Here is that pesky {xxx/2:.3f} again")
           ^^^
NameError: name 'xxx' is not defined

    Removing the formatting part ":.3f" makes it work as expected

    @pablogsal pablogsal added 3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error labels Aug 11, 2021
    @gvanrossum
    Copy link
    Member

    (I originally reported this.)

    @pablogsal
    Copy link
    Member Author

    The problem lies here:

    cpython/Parser/string_parser.c

    Lines 374 to 389 in f66d00f

    // The call to fstring_find_expr_location is responsible for finding the column offset
    // the generated AST nodes need to be shifted to the right, which is equal to the number
    // of the f-string characters before the expression starts. In order to correctly compute
    // this offset, strstr gets called in fstring_find_expr_location which only succeeds
    // if curly braces appear before and after the f-string expression (exactly like they do
    // in the f-string itself), hence the following lines.
    str[0] = '{';
    memcpy(str+1, expr_start, len);
    str[len+1] = '}';
    str[len+2] = 0;
    int lines, cols;
    if (!fstring_find_expr_location(t, str, &lines, &cols)) {
    PyMem_Free(str);
    return NULL;
    }

    The problem is that the strstr call will fail because the string containing the expression doesn't have the formatting part in it, so the offsets never get corrected.

    @pablogsal
    Copy link
    Member Author

    Actually, this has even more problems. Because we are using strstr to find the start of the expression in the parent string, if the expression is repeated the offsets are incorrectly generated:

    For example:

    print(f"Here is that {xxx} pesky {xxx} again")

    This produces:

    ...
    FormattedValue(
    value=Name(
    id='xxx',
    ctx=Load(),
    lineno=1,
    col_offset=22,
    end_lineno=1,
    end_col_offset=25),
    ...
    FormattedValue(
    value=Name(
    id='xxxx',
    ctx=Load(),
    lineno=1,
    col_offset=22,
    end_lineno=1,
    end_col_offset=25),
    ...

    while

    print(f"Here is that {xxx} pesky {xxxx} again")

    (different variables) produces:

    ...
    FormattedValue(
    value=Name(
    id='xxx',
    ctx=Load(),
    lineno=1,
    col_offset=22,
    end_lineno=1,
    end_col_offset=25),
    ...
    FormattedValue(
    value=Name(
    id='xxxx',
    ctx=Load(),
    lineno=1,
    col_offset=34,
    end_lineno=1,
    end_col_offset=38),
    ...

    @pablogsal
    Copy link
    Member Author

    Ha, we have test that knows is broken:

    cpython/Lib/test/test_fstring.py

    Line 217 in f66d00f

    NOTE: this is currently broken, always sets location of the first

    @pablogsal
    Copy link
    Member Author

    New changeset 8e832fb by Pablo Galindo Salgado in branch 'main':
    bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729)
    8e832fb

    @pablogsal
    Copy link
    Member Author

    New changeset c28c2e1 by Pablo Galindo Salgado in branch '3.10':
    [3.10] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27743)
    c28c2e1

    @pablogsal
    Copy link
    Member Author

    New changeset 4b86c9c by Pablo Galindo Salgado in branch '3.9':
    [3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27744)
    4b86c9c

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @wookayin wookayin mentioned this issue Sep 18, 2022
    Open
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.11 only security fixes interpreter-core (Objects, Python, Grammar, and Parser dirs) type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @gvanrossum @pablogsal