Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ast.literal_eval does not accept strings with leading whitespaces #86053

Closed
gousaiyang mannequin opened this issue Sep 30, 2020 · 8 comments
Closed

ast.literal_eval does not accept strings with leading whitespaces #86053

gousaiyang mannequin opened this issue Sep 30, 2020 · 8 comments
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@gousaiyang
Copy link
Mannequin

gousaiyang mannequin commented Sep 30, 2020

BPO 41887
Nosy @gvanrossum, @terryjreedy, @pablogsal, @miss-islington, @isidentical, @gousaiyang
PRs
  • bpo-41887: omit leading whitespaces on ast.literal_eval #22469
  • [3.9] bpo-41887: omit leading spaces/tabs on ast.literal_eval (GH-22469) #22533
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2020-10-04.00:56:32.242>
    created_at = <Date 2020-09-30.01:46:57.482>
    labels = ['3.8', 'type-feature', 'library', '3.9', '3.10']
    title = 'ast.literal_eval does not accept strings with leading whitespaces'
    updated_at = <Date 2020-10-04.00:56:32.241>
    user = 'https://github.com/gousaiyang'

    bugs.python.org fields:

    activity = <Date 2020-10-04.00:56:32.241>
    actor = 'gvanrossum'
    assignee = 'none'
    closed = True
    closed_date = <Date 2020-10-04.00:56:32.242>
    closer = 'gvanrossum'
    components = ['Library (Lib)']
    creation = <Date 2020-09-30.01:46:57.482>
    creator = 'gousaiyang'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 41887
    keywords = ['patch']
    message_count = 8.0
    messages = ['377687', '377711', '377722', '377843', '377853', '377882', '377909', '377911']
    nosy_count = 6.0
    nosy_names = ['gvanrossum', 'terry.reedy', 'pablogsal', 'miss-islington', 'BTaskaya', 'gousaiyang']
    pr_nums = ['22469', '22533']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue41887'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @gousaiyang
    Copy link
    Mannequin Author

    gousaiyang mannequin commented Sep 30, 2020

    ast.literal_eval does not accept code with leading whitespaces, while eval accepts them, which is an inconsistency.

    >>> import ast
    >>> eval(' 1')
    1
    >>> ast.literal_eval(' 1')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/local/lib/python3.9/ast.py", line 62, in literal_eval
        node_or_string = parse(node_or_string, mode='eval')
      File "/usr/local/lib/python3.9/ast.py", line 50, in parse
        return compile(source, filename, mode, flags,
      File "<unknown>", line 1
        1
    IndentationError: unexpected indent
    

    @gousaiyang gousaiyang mannequin added 3.7 (EOL) end of life 3.8 only security fixes 3.9 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Sep 30, 2020
    @isidentical
    Copy link
    Sponsor Member

    Looks like skipping the leading whitespace is an undocumented behavior for eval() introduced back in 1992 (f08ab0a). I don't think that bringing it to the literal_eval is a good idea, at least after this much of a time. If so, both of them should be explicitly documented or at least mentioned in the source code (to avoid confusion).

    @gvanrossum
    Copy link
    Member

    Hm, I'm not sure. ast.literal_eval() does accept trailing whitespace, and embedded whitespace.

    >>> ast.literal_eval("- ( 1\n)\n")
    -1
    >>> 
    

    So I think it should start accepting leading whitespace too (but only in a feature release, so 3.10).

    @terryjreedy
    Copy link
    Member

    The doc for literal_eval says "evaluate ... a string containing a Python literal or container display." To me, ' 1' qualifies, just as it does as an expression for eval().

    The exception comes from parsing raising IndentationError with leading whitespace even when the mode is 'eval' rather than 'exec'. This surprised me. Eval() gets strips the beginning of the string before it is parsed. If parsing remains as is, I agree that doing the same for literal_eval strings. But why should not parsing remove indents for 'eval' mode?

    @terryjreedy terryjreedy added 3.10 only security fixes and removed 3.7 (EOL) end of life labels Oct 3, 2020
    @gvanrossum
    Copy link
    Member

    [Terry]

    But why should not parsing remove indents for 'eval' mode?

    I dunno, but it's been doing this since 1992, so I think it would be fragile to change. The best thing therefore is to make ast.literal_eval() match it exactly.

    @pablogsal
    Copy link
    Member

    I dunno, but it's been doing this since 1992, so I think it would be fragile to change. The best thing therefore is to make ast.literal_eval() match it exactly.

    +1 We had considerable finicky behaviour in the parser related to new lines and whitespace so I would recommend to strive for stability.

    @gvanrossum
    Copy link
    Member

    New changeset e799aa8 by Batuhan Taskaya in branch 'master':
    bpo-41887: omit leading spaces/tabs on ast.literal_eval (bpo-22469)
    e799aa8

    @gvanrossum
    Copy link
    Member

    Closing. Let’s not backport.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants