Author David Bieber
Recipients David Bieber
Date 2017-10-12.22:31:18
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1507847478.46.0.213398074469.issue31778@psf.upfronthosting.co.za>
In-reply-to
Content
# Overview
ast.literal_eval supports some non-literals in Python 3.
The behavior of this function runs contrary to the documented behavior.


# The Issue
The [documentation](https://docs.python.org/3/library/ast.html#ast.literal_eval) says of the function "It is not capable of evaluating arbitrarily complex expressions, for example involving operators or indexing."

However, literal_eval is capable of evaluating expressions with certain operators, particular the operators "+" and "-".

As has been explained previously, the reason for this is to support "complex" literals such as 2+3j. However, this has unintended consequences which I believe to be indicative of a bug. Examples of the unintended consequences include `ast.literal_eval('1+1') == 2` `ast.literal_eval('2017-10-10') == 1997`. I would expect each of these calls to literal_eval to throw a ValueError, as the input string is not a literal. Instead, literal_eval successfully evaluates the input string, in the latter case giving an unexpected result (since the intent of the string is to represent a 21st century date.)

Since issue arose as a [Python Fire issue](https://github.com/google/python-fire/issues/97), where the behavior of Python Fire was unexpected for inputs such as those described above (1+1 and 2017-10-10) only in Python 3. For context, Python Fire is a CLI library which uses literal_eval as part of its command line argument parsing procedure.

I think the resolution to this issue is having literal_eval raise a ValueError if the ast of the input represents anything other than a Python literal, as described in the documentation. That is, "The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None." Additional operations, such as the binary operations "+" and "-", unless they explicitly create a complex number, should produce a ValueError.

If that resolution is not the direction we take, I also would appreciate knowing if there is another built in approach for determining if a string or ast node represents a literal.


# Reproducing
The following code snippets produce different behaviors in Python 2 and Python 3.
```python
import ast
ast.literal_eval('1+1')
```

```python
import ast
ast.literal_eval('2017-10-10')
```


# References
- The Python Fire issue is here: https://github.com/google/python-fire/issues/97
- Prior discussion of a similar issue: https://bugs.python.org/issue22525
- I think is where complex number support was originally added: https://bugs.python.org/issue4907
- In this thread, https://bugs.python.org/issue24146, one commenter explains that literal_eval's support for "2+3" is an unintentional side effect of supporting complex literals.
History
Date User Action Args
2017-10-12 22:31:18David Biebersetrecipients: + David Bieber
2017-10-12 22:31:18David Biebersetmessageid: <1507847478.46.0.213398074469.issue31778@psf.upfronthosting.co.za>
2017-10-12 22:31:18David Bieberlinkissue31778 messages
2017-10-12 22:31:18David Biebercreate