This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: mismatched eval() and ast.literal_eval() behavior with unicode_literals
Type: enhancement Stage: resolved
Components: Documentation Versions:
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, jez, steven.daprano
Priority: normal Keywords:

Created on 2019-01-18 21:26 by jez, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (3)
msg334011 - (view) Author: jez (jez) Date: 2019-01-18 21:26
Following  `from __future__ import unicode_literals`   the expression `eval(" 'foo' ")` will return a `unicode` instance.  However, using the same input, `ast.literal_eval(" 'foo' ")` will return a `str` instance. The caller's preference, that those plain single-quotes should a denote unicode literal, is respected by `eval()` but not by `ast.literal_eval()`.   I propose that `ast.literal_eval()` be made sensitive to this preference, to bring it in line with `eval()`.
msg334015 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-01-18 22:03
Python 2.7 has long passed feature freeze, and this would be new behaviour appearing in a bug-fix release, which we don't normally do.

I'm going to close this as Rejected, but if you think you can make a good case for why this enhancement is sufficiently important to break backwards-compatibility in a bug-fix release, or alternatively that the current behaviour is a bug (e.g. that it goes against the documentation) then we can re-open it.

(Merely "doesn't do what I think the user expects" is not a bug.)
msg334161 - (view) Author: jez (jez) Date: 2019-01-21 18:15
I take the point about breaking backward compatibility. The problem, then, is in the `ast` documentation, and may not be confined to Python 2.7

The `literal_eval` doc <https://docs.python.org/2/library/ast.html#ast-helpers> states no more than that the function evaluates a "string containing a Python literal".  The future statement effectively changes the definition of "a Python literal" but of course, as the official `__future__` doc <https://docs.python.org/2/reference/simple_stmts.html#future> explains, this is done for the compilation of each "particular module" in isolation. The question is left open whether the occurrence of this term in `ast` documentation should be read as "a Python literal (as defined by the ast module itself at compile time)",  or "a Python literal (as defined by the caller at compile time)".

More generally, future statements effectively change the definition of the term "Python syntax", sometimes also simply referred to as "Python".  Unfortunately, the `ast` documentation uses these terms loosely throughout, without acknowledging that they are mutable. I propose that the `ast` module documentation flag cases in which the `ast` module's own definitions of "Python" and "Python syntax" may not match those of a caller who has included `__future__` statements, leading to unexpected behavior.

I am insufficiently familiar with the full functionality of `ast` to be able to identify where this is and is not an issue, except in `literal_eval`, but I can see it could in principle affect Python 3.x too.
History
Date User Action Args
2022-04-11 14:59:10adminsetgithub: 79958
2019-01-21 18:15:53jezsetversions: - Python 2.7
nosy: + docs@python

messages: + msg334161

assignee: docs@python
components: + Documentation
2019-01-18 22:03:26steven.dapranosetstatus: open -> closed

type: behavior -> enhancement

nosy: + steven.daprano
messages: + msg334015
resolution: rejected
stage: resolved
2019-01-18 21:26:40jezcreate