classification
Title: ast.literal_eval() doesn't do what the documentation says
Type: enhancement Stage: resolved
Components: Documentation Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Behdad.Esfahbod, docs@python, ezio.melotti, georg.brandl, python-dev, r.david.murray, rhettinger
Priority: normal Keywords: patch

Created on 2014-09-30 14:57 by Behdad.Esfahbod, last changed 2014-11-05 19:27 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
ast_literal_eval_clarify.patch georg.brandl, 2014-09-30 17:44 review
Messages (12)
msg227941 - (view) Author: Behdad Esfahbod (Behdad.Esfahbod) Date: 2014-09-30 14:57
The documentation says:

"""
Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and None.

This can be used for safely evaluating strings containing Python expressions from untrusted sources without the need to parse the values oneself.
"""

This makes me to believe that this is a useful replacement for eval() that is safe.  However, it fails to make it clear that it parses **one literal**, NOT an expression.  Ie. it can't handle "2*2".  Weirdly enough, at least with my Python 3.2.3, it does handle "2+2" with no problem.

This seriously limits the usefulness of this function.  Is there really no equivalent that parses simple expressions of literals?
msg227956 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-09-30 17:29
I agree that the wording can be improved.  The term "expression" is used here loosely, but should be clarified.

"a+b" with numbers a, b must be handled in order to evaluate complex literals as in "1+1j".

Allowing operator expressions with literals is possible, but much more complex than the current implementation.  A simple implementation is not safe: you can induce basically unbounded CPU and memory usage with no effort (try "9**9**9" or "[None] * 9**9").

As for the usefulness, this function is useful to "read back" literal values and containers as stringified by repr().  This can for example be used for serialization in a format that is similar to but more powerful than JSON.
msg227963 - (view) Author: Behdad Esfahbod (Behdad.Esfahbod) Date: 2014-09-30 17:37
I think it should be made much more clear that this is not a blanket "safe eval() replacement".

Re complex literals, note that Python 2.7.x only implemented the binary plus operator if the second argument was complex.  This seems to have been relaxed in Python 3.

Regarding DoS attack with a safe eval(), I understand the concern, but that's still a huge improvement over security risks of eval().
msg227965 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-09-30 17:40
The function is still called literal_eval(), not safe_eval().

I'm not saying a safe eval() isn't useful.  But an implementation that is only partly safe is not going to help anyone.
msg227966 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-09-30 17:44
Attaching proposed doc change.
msg227967 - (view) Author: Behdad Esfahbod (Behdad.Esfahbod) Date: 2014-09-30 17:47
Thanks.  In your proposed text:

+ Safely evaluate an expression node or a string containing a Python literal or container display.

I suggest changing it to "...containing a single Python literal or..."

Ie, adding "single".
msg227986 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-09-30 19:33
It should probably also say "or a container display containing only literals".  (That's a lot of "contain" :)

(I had the same confusion the first time I read these docs, BTW).
msg227988 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-09-30 19:40
Sure, feel free to reword.

An example or two might also be helpful.
msg230670 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-11-05 08:49
Georg's proposed wording reads well and is clearer than the current wording.  The patch is ready to apply.
msg230706 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-11-05 19:21
New changeset 5c5909740026 by Georg Brandl in branch '3.4':
Closes #22525: clarify documentation for ast.literal_eval().
https://hg.python.org/cpython/rev/5c5909740026
msg230707 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2014-11-05 19:24
Thanks, Raymond.
msg230708 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-11-05 19:27
New changeset 3e8d3c4bc17e by Georg Brandl in branch '2.7':
Closes #22525: clarify documentation for ast.literal_eval().
https://hg.python.org/cpython/rev/3e8d3c4bc17e
History
Date User Action Args
2014-11-05 19:27:22python-devsetmessages: + msg230708
2014-11-05 19:24:25georg.brandlsetmessages: + msg230707
2014-11-05 19:21:44python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg230706

resolution: fixed
stage: patch review -> resolved
2014-11-05 08:49:54rhettingersetnosy: + rhettinger
messages: + msg230670
2014-11-02 15:47:05ezio.melottisetnosy: + ezio.melotti

type: enhancement
stage: patch review
2014-09-30 19:40:24georg.brandlsetmessages: + msg227988
2014-09-30 19:33:20r.david.murraysetnosy: + r.david.murray
messages: + msg227986
2014-09-30 17:47:32Behdad.Esfahbodsetmessages: + msg227967
2014-09-30 17:44:25georg.brandlsetfiles: + ast_literal_eval_clarify.patch
keywords: + patch
messages: + msg227966
2014-09-30 17:40:31georg.brandlsetmessages: + msg227965
2014-09-30 17:37:15Behdad.Esfahbodsetmessages: + msg227963
2014-09-30 17:29:54georg.brandlsetnosy: + georg.brandl
messages: + msg227956
2014-09-30 16:39:31pitrousetassignee: docs@python

components: + Documentation
nosy: + docs@python
2014-09-30 14:57:19Behdad.Esfahbodcreate