This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: ast module is combining string literals that are concatenated with plus operator
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder: ast Str type does not annotate the string type when it parses a python document
View: 25885
Assigned To: Nosy List: SilentGhost, benjamin.peterson, brett.cannon, georg.brandl, myronww, ncoghlan, r.david.murray, vstinner
Priority: normal Keywords:

Created on 2015-12-16 21:45 by myronww, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (6)
msg256533 - (view) Author: Myron Walker (myronww) * Date: 2015-12-16 21:45
ast module is combining string literals that are concatenated with plus operator into a single string in the parsestrplus module rather than maintaining the true syntax tree of the original document.

The fact that ast combines strings and does not maintain the original syntax of the document means that original syntax information is lost and a writer walking the abstract syntax tree is unable to reproduce the original syntax
of the document.
msg256537 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2015-12-16 21:50
Hm, I'm going to close this as duplicate since a commenter in your previous issue clearly stated: "I don't think the AST module makes any guarantee about being able to reproduce the original source".
msg256540 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-12-16 21:52
Also note that this probably mirrors the behavior of the actual compiler (I haven't checked).
msg256541 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-12-16 21:54
By the way, if you want to talk about enhancing the AST module to be more useful to your use cases, the python-ideas mailing list would be the right place to raise the topic (you might want to look for earlier relevant discussions first, I'm pretty sure there have been a couple).
msg256542 - (view) Author: Myron Walker (myronww) * Date: 2015-12-16 21:58
My thought on this is that a syntax tree needs to accurately represent parsed syntax of the code.  Two strings being concatenated contain syntax information that is different from a single string.

  "Hello" + " World"

Is not the same syntax as:

  "Hello World"

To lose that syntax information seems to defeat the intent of abstract syntax tree which is to store the syntax associated with a document.
msg256543 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-12-16 21:58
"The fact that ast combines strings and does not maintain the original syntax of the document means that original syntax information is lost and a writer walking the abstract syntax tree is unable to reproduce the original syntax of the document."

You should take a look at the RedBaron project which implements a Python parser, provide a high-level representation of the code *and* allow to regenerates the Python code without loosing the indentation, comment, or anything else.

https://redbaron.readthedocs.org/en/latest/

"To achieve this, [RedBaron] is based on Baron a lossless AST for Python that guarantees the operation
fst_to_code(code_to_fst(source_code)) == source_code."
History
Date User Action Args
2022-04-11 14:58:25adminsetgithub: 70074
2015-12-16 23:23:00brett.cannonsetresolution: duplicate -> not a bug
2015-12-16 21:58:20vstinnersetnosy: + vstinner
messages: + msg256543
2015-12-16 21:58:16myronwwsetmessages: + msg256542
2015-12-16 21:54:14r.david.murraysetmessages: + msg256541
2015-12-16 21:52:59r.david.murraysetnosy: + r.david.murray
messages: + msg256540
2015-12-16 21:50:50SilentGhostsetstatus: open -> closed

superseder: ast Str type does not annotate the string type when it parses a python document
nosy: + brett.cannon, ncoghlan, benjamin.peterson, georg.brandl, SilentGhost

messages: + msg256537
type: behavior
resolution: duplicate
stage: resolved
2015-12-16 21:45:04myronwwcreate