classification
Title: ast.literal_eval does not accept strings with leading whitespaces
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, gousaiyang, gvanrossum, miss-islington, pablogsal, terry.reedy
Priority: normal Keywords: patch

Created on 2020-09-30 01:46 by gousaiyang, last changed 2020-10-04 00:56 by gvanrossum. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22469 merged BTaskaya, 2020-09-30 20:52
PR 22533 closed miss-islington, 2020-10-04 00:49
Messages (8)
msg377687 - (view) Author: Saiyang Gou (gousaiyang) * Date: 2020-09-30 01:46
`ast.literal_eval` does not accept code with leading whitespaces, while `eval` accepts them, which is an inconsistency.

```
>>> import ast
>>> eval(' 1')
1
>>> ast.literal_eval(' 1')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/ast.py", line 62, in literal_eval
    node_or_string = parse(node_or_string, mode='eval')
  File "/usr/local/lib/python3.9/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    1
IndentationError: unexpected indent
```
msg377711 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python committer) Date: 2020-09-30 15:58
Looks like skipping the leading whitespace is an undocumented behavior for eval() introduced back in 1992 (f08ab0ad158f88f05dd923b129d2397e1882be14). I don't think that bringing it to the literal_eval is a good idea, at least after this much of a time. If so, both of them should be explicitly documented or at least mentioned in the source code (to avoid confusion).
msg377722 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-30 20:33
Hm, I'm not sure. ast.literal_eval() does accept trailing whitespace, and embedded whitespace.

```
>>> ast.literal_eval("- ( 1\n)\n")
-1
>>> 
```

So I think it should start accepting leading whitespace too (but only in a feature release, so 3.10).
msg377843 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-10-03 00:25
The doc for literal_eval says "evaluate ... a string containing a Python literal or container display."  To me, ' 1' qualifies, just as it does as an expression for eval().

The exception comes from parsing raising IndentationError with leading whitespace even when the mode is 'eval' rather than 'exec'.  This surprised me. Eval() gets strips the beginning of the string before it is parsed.  If parsing remains as is, I agree that doing the same for literal_eval strings.  But why should not parsing remove indents for 'eval' mode?
msg377853 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-10-03 05:10
[Terry]
> But why should not parsing remove indents for 'eval' mode?

I dunno, but it's been doing this since 1992, so I think it would be fragile to change. The best thing therefore is to make ast.literal_eval() match it exactly.
msg377882 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-10-03 14:32
> I dunno, but it's been doing this since 1992, so I think it would be fragile to change. The best thing therefore is to make ast.literal_eval() match it exactly.

+1 We had considerable finicky behaviour in the parser related to new lines and whitespace so I would recommend to strive for stability.
msg377909 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-10-04 00:46
New changeset e799aa8b92c195735f379940acd9925961ad04ec by Batuhan Taskaya in branch 'master':
bpo-41887: omit leading spaces/tabs on ast.literal_eval (#22469)
https://github.com/python/cpython/commit/e799aa8b92c195735f379940acd9925961ad04ec
msg377911 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-10-04 00:56
Closing. Let’s not backport.
History
Date User Action Args
2020-10-04 00:56:32gvanrossumsetstatus: open -> closed
resolution: fixed
messages: + msg377911

stage: patch review -> resolved
2020-10-04 00:49:00miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request21536
2020-10-04 00:46:47gvanrossumsetmessages: + msg377909
2020-10-03 14:32:49pablogsalsetmessages: + msg377882
2020-10-03 05:10:35gvanrossumsetmessages: + msg377853
2020-10-03 00:25:01terry.reedysetnosy: + terry.reedy

messages: + msg377843
versions: + Python 3.10, - Python 3.6, Python 3.7
2020-09-30 20:52:36BTaskayasetkeywords: + patch
stage: patch review
pull_requests: + pull_request21492
2020-09-30 20:33:49gvanrossumsetmessages: + msg377722
2020-09-30 20:18:40rhettingersetnosy: + gvanrossum
2020-09-30 15:58:30BTaskayasetnosy: + pablogsal
messages: + msg377711
2020-09-30 15:46:25BTaskayasetnosy: + BTaskaya
2020-09-30 01:46:57gousaiyangcreate