The 'ast' module does not indicate the type of string, ''' or '"' or '"""', that it has encountered when it parses a python document.

This prevents accurate reproduction of the original parsed document by a writer walking over an instance of a abstract syntax tree.

Two things. One, this won't change in Python 2.7 due to compatibility, so this only applies to Python 3.6. Two, the AST has not been designed to support round-trip syntax transpiling, e.g. there is no way to get back comments in the source code.

Because the AST is not meant to round-trip on source code I'm going to close this as "not a bug" since I don't see any benefit in supporting this even if we gain comment nodes in the AST.

Unlike the tokenizer, I don't think the AST module makes any guarantee about being able to reproduce the original source. This might be a reasonable enhancement request, though.

Oh, that was weird. I got a weird error message from roundup. Must have been because I was posting at the same time as Brett. I'll defer to his decision on this.

The purpose of a syntax tree is to represent the syntax and not a final processed result of processing of syntax. The current information stored for strings is losing syntax information which seems to defeat the purpose of offering the information in a syntax tree. I filed a separate bug because it is also combining strings and losing operators for string literals.

"Hello" + " World"

From the looks of the code, the above would result in one string type with "Hello World" and syntax information associated with the operator would be lost.

And as indicated, string type information is being lost as well. The user of the AST then has no way of getting the lost syntax information back once it is lost.

I am re-opening this as I believe this is an important issue for work I would like to eventually push into the python core which is python code that recode themselves as declarations or as instance representation.

"I don't see any benefit in supporting this even if we gain comment nodes in the AST."

There would be a huge benefit to having accurate original syntax information of the document and being able to re-write the documents as it would enable self modifying code or software written code paradigms.

TestCases that can update themselves in various modes, full auto, interactive, etc.

TestData Pattern Generation object or Data Pattern Generation object that can be easily or efficiently modified and then asked to write themselves back out to files. 

The fact that there are other projects being worked on to provide round trip document to syntax tree and syntax tree to document transformations also indicates that the lack of accurate syntax storage in 'ast' is a problem that
needs to be addressed.

I would prefer to work with the core python modules in order to provide dynamic code modification functionality rather than relying on a third party module as eventually I would like to push this into core python.

If a python object has script behind it, I would like to eventually be able to tell a the object to write itself to a file as a object declaration or as a object instance.

As Declaration:

class SomeObject(SomeBase):
   def some method(self):
       print "blah blah"

As Instance:

   SomeObject()

Please respect the decisions of the python core developers as to issue status (you can argue with us until we ask you to stop, but let us make the status change if you convince us :)

We aren't rejecting your ideas, just this bug report (as not being a bug). To pursue your ideas you will want to subscribe to the python-ideas mailing list and begin a discussion there. This is far too big a topic for the bug tracker.

"There would be a huge benefit to having accurate original syntax information of the document and being able to re-write the documents as it would enable self modifying code or software written code paradigms."

You misunderstood the purpose of the AST tree. "Unparse" AST to retrieve the original source code is out of the scope of the Python AST structure and it's not going to happen. For that, you must use other tool. See for example RedBaron, as I said at:

http://bugs.python.org/issue25886#msg256543

Being able to "unparse" AST to regenerate the Python original source code requires complex operation. You have to store all formating information (spaces, new lines, etc.). You also have to store comments. It's just to give a few examples.

If AST contains formating information, it would be more complex to handle AST.

Why is unparsing out of the scope of the base syntax tree of a dynamic language such as python?

Is that just a personal opinion? Would adding proper storage of syntax information in the AST cause performance issues?

Is there some documentation that defines the scope of the AST module that indicates this is out of scope.

The syntax for numerical constants is properly stored by the AST module.

Example:
TEST_INT = 1 + 1

Why would the syntax of integer constants be more important than the syntax associated with comments, two string literals, or from multi-line strings.

"Would adding proper storage of syntax information in the AST cause performance issues?"

Exactly. And storing syntax information is useless for 90% of usages of AST. So Python is optimized for the most common cases.

Again, use a different project (like RedBaron) if you need *all* information in the AST.

Would it be prudent to add a Parse flag to the AST module that could provide one of two types of AST's an optimized AST or a complete AST

Would it be prudent to add a Parse flag to the AST module that could provide one of two types of AST's an optimized AST or a complete AST

Adding syntax ("formatting", call it as you want) info to AST requires
to add new attributes to existing AST nodes and probably add new AST
nodes. It will make the code more complex, not only the code to
produce AST, but also code using AST (static code analyzer, my AST
optimizer project, etc.).

I don't think that it's worth it.

If you consider that we are wrong, please follow the advice of
starting a discussion on python-ideas. This is how Python is
developed, we have a workflow. Proposing ideas on the bug tracker
works in some cases, but AST is really a *core* feature of Python. You
need deep discussions to change the core. To be cristal clear: *if*
anyone is going to change AST, IMHO a PEP is needed. Writing a PEP
requires a specific workflow starting at python-ideas.

I agree that the proposed change would require a PEP, and this should be discussed on python-ideas.

I also think there's very little chance such a change would be accepted, but that doesn't mean it's impossible.

I think using a external library is your best bet here. If you want to pursue this on python-ideas, you should discuss why being in the stdlib's ast module would be an improvement over using an external library.