Title: ast.literal_eval fails on some literals
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.4
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: JBernardo, amaury.forgeotdarc, benjamin.peterson, daniel.urban, eric.araujo, ezio.melotti, georg.brandl, mark.dickinson, serhiy.storchaka
Priority: normal Keywords: needs review, patch

Created on 2012-07-03 20:11 by JBernardo, last changed 2018-11-21 07:51 by serhiy.storchaka. This issue is now closed.

File name Uploaded Description Edit JBernardo, 2012-07-05 21:00 Added support to `...` and a mapping for safe names as second argument. review
ast.py_with_tests.diff JBernardo, 2012-07-06 00:02 Changed docstring and added tests. review
ast.py_tests_doc.diff JBernardo, 2012-07-06 00:47 Now with doc update. review
Messages (14)
msg164621 - (view) Author: João Bernardo (JBernardo) * Date: 2012-07-03 20:11
`ast.literal_eval` is very strict on names, so it seems to lack some "literals" that may be result of `repr` on built-in objects.

-> Obvious cases:


both result on ValueError.

-> Not so obvious:

nan_name = repr(float('nan'))
ast.literal_eval(nan_name) # ValueError

inf_name = repr(float('inf'))
ast.literal_eval(inf_name)  # ValueError

ast.literal_eval("2e308") # == inf

`nan` and `inf` are not literals (at least "inf" should be, but that's another problem), but their representations are not possible to be evaluated unlike any other float numbers with maybe precision loss. 

I think `literal_eval` should include these 3 names the same way it accepts True, False and None.

Another case, that I personally don't care, but seems plausible would be `NotImplemented`.
msg164651 - (view) Author: Daniel Urban (daniel.urban) * (Python triager) Date: 2012-07-04 16:22
eval(repr(float('nan'))) doesn't work either, so I'm not sure ast.literal_eval should. The same with float('inf').
msg164658 - (view) Author: João Bernardo (JBernardo) * Date: 2012-07-04 18:44
That's what I said by `nan` and `inf` are not literals, but their representations look like they should be.

One solution could be to add another argument to allow some extra names. Maybe a mapping, as `eval`.
msg164666 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2012-07-05 01:54
I don't think literal_eval should handle anything which is not syntax.
msg164679 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-07-05 12:05
> I don't think literal_eval should handle anything which is not syntax.

-1 on supporting 'inf' and 'nan'
-0 on supporting 'Ellipsis' or '...';  seems harmless, but I don't really see the point.
msg164684 - (view) Author: João Bernardo (JBernardo) * Date: 2012-07-05 15:48
Of course `nan` and `inf` are part of the syntax! The `ast.parse` function recognize them as `Name`. 

So that works:

>>> ast.dump(ast.parse('True'))
"Module(body=[Expr(value=Name(id='True', ctx=Load()))])"

>>> ast.dump(ast.parse('inf'))
"Module(body=[Expr(value=Name(id='inf', ctx=Load()))])"

>>> inf = float('inf')
>>> eval('inf')

I've run into some literals with `Ellipsis` and `inf` and couldn't load them with literal_eval. That's why I'm proposing that.

The thing about `nan` and `inf` is because they are the *only* representations of float numbers produced by the interpreter that cannot be loaded.

Something like that could solve the problem keeping `literal_eval` safe and allowing other names:

ast.literal_eval('[1.0, 2.0, inf]', {'inf': float('inf')})
msg164698 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-05 22:10
I like the 'safe_names' idea, but is this patch supposed to work?

>>> isinstance(None, Ellipsis)
TypeError: isinstance() arg 2 must be a type or tuple of types
msg164699 - (view) Author: João Bernardo (JBernardo) * Date: 2012-07-05 22:12
Ellipsis in this context is `_ast.Ellipsis`, not the original one...

There's no TypeError there as `_ast.Ellipsis` is a class.
msg164700 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-07-05 22:19
Ah right, that's confusing indeed...

Unit tests and doc updates are needed though.
msg164707 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-06 00:21
Thanks for the patch.  It looks good to me, but I think that adding support for Ellipsis and adding a new parameter to allow extra names are two different requests, but someone more knowledgeable than me about AST may judge differently.  My rationale is that supporting Ellipsis literals is a no-brainer addition, whereas adding an argument to support things that are not actual Python literals may be rejected.  You don’t have to split your patch in two anyway, it’s easy to do if the dev who will commit wants to.

Each change would also need a doc update.
msg164708 - (view) Author: João Bernardo (JBernardo) * Date: 2012-07-06 00:29
Just to be sure: What's a doc update? The `.rst` files are updated automatically with the doc strings?

The adding another argument should require a:
"Changed in version 3.x: Accepts a second argument ..." ?
msg164709 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-06 00:32
No, the rst documentation is maintained manually.

Look for versionchanged in the doc source files or the documentation about writing documentation to see how to document the new argument (and new supported Ellipsis literal).
msg329284 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-05 07:55
ast.literal_eval('...') works in 3.8.

The question is should we make it working in older versions?
msg330040 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2018-11-18 10:05
> The question is should we make it working in older versions? 

That seems like an easy question: it would be a new feature in a bugfix branch, so no, we shouldn't make it work in older versions.

Looks to me as though this issue can be closed.
Date User Action Args
2018-11-21 07:51:19serhiy.storchakasetstatus: open -> closed
resolution: out of date
stage: patch review -> resolved
2018-11-18 10:05:40mark.dickinsonsetmessages: + msg330040
2018-11-05 07:55:19serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg329284
2012-07-06 17:29:31ezio.melottisetnosy: + ezio.melotti
2012-07-06 00:47:59JBernardosetfiles: + ast.py_tests_doc.diff
2012-07-06 00:32:14eric.araujosetmessages: + msg164709
2012-07-06 00:29:40JBernardosetmessages: + msg164708
2012-07-06 00:21:05eric.araujosetmessages: + msg164707
stage: test needed -> patch review
2012-07-06 00:02:59JBernardosetfiles: + ast.py_with_tests.diff
2012-07-05 22:19:11amaury.forgeotdarcsetmessages: + msg164700
2012-07-05 22:12:40JBernardosetmessages: + msg164699
2012-07-05 22:10:49amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg164698
2012-07-05 21:02:32eric.araujosetkeywords: + needs review
stage: test needed
2012-07-05 21:00:35JBernardosetfiles: +
keywords: + patch
2012-07-05 16:37:59eric.araujosetnosy: + georg.brandl, eric.araujo
2012-07-05 15:48:05JBernardosetmessages: + msg164684
2012-07-05 12:05:41mark.dickinsonsetmessages: + msg164679
2012-07-05 01:54:31benjamin.petersonsetmessages: + msg164666
2012-07-04 18:44:31JBernardosetmessages: + msg164658
2012-07-04 16:22:02daniel.urbansetnosy: + daniel.urban
messages: + msg164651
2012-07-04 14:47:08mark.dickinsonsetnosy: + mark.dickinson
2012-07-03 20:54:32pitrousetnosy: + benjamin.peterson
title: ast.literal_eval on some literals -> ast.literal_eval fails on some literals

versions: + Python 3.4, - Python 3.3
2012-07-03 20:11:26JBernardocreate