Hi,

The following code gives similar errors for both PyPy and Python 2 when called, but totally different AST objects.

Given the following code, PyPy says that `print(4)` is an _ast.Print, while Python 2 believes that it is an _ast.Expr (which wraps a Call). This behaviour is wrong, Python 2 shouldn't extrapolate further if the __future__ is misplaced.


   from ast import parse

   bad = """
   a = 1
   from __future__ import print_function
   print(4)
   """

   x = parse(bad)
   print(x.body[2])

Brett, I added you on this as nosy, since I discovered this bug while debugging your print-statement pull request, so I thought you should know. :-)

A difference in internal behavior between CPython and PyPy is not necessarily a CPython bug, or even a PyPy bug.  It may just be an implementation choice.  So ignore PyPy and consider Python doc versus CPython behavior.

Running the three quoted lines by themselves gives SyntaxError.  compiling the ast object does the same.  So this is one of those SyntaxErrors not noticed (or at least not enforced) until the compile phase. This was specified in PEP 236 and still is in the Reference: "A future statement is recognized and treated specially at compile time".  Whenever the compiler raises SyntaxError, it declares the AST to be syntactically invalid.  How can this be?

The ast doc starts with "The ast module helps Python applications to process trees of the Python *abstract syntax grammar*. ... this module helps to find out programmatically what the current grammar looks like." (asterisks added). The abstract grammar does *not* include a 'Future' statement. In your code, x.body[1] is an ImportFrom object.  So the ast from the code above does not violate the abstract grammar and there is no parsing bug.

We should either either close this as 'not a bug' or change to a doc issue to add something about the ast abstract grammar being 'looser' than the defacto context-sensitive Python grammar.  Perhaps something like "The language defined by the context-free LL(1) abstract grammer is a superset of the actual context-sensitive Python language. (Consider position-dependent future statements.)  Consequently, some ast trees generate a SyntaxError when compiled."

It is somewhat arbitrary when a print statement is converted to a print expression -- while parsing, or while compiling.  The two implementations made different choices.

Since the parser does recognize a Future statement and change the ast objects it creates, it would be possible to add a Future object to ast trees.  But it would do no good as far as the syntax check goes.  The context-free LL(1) parser could not recognize a violation of the context-sensitive constraint.

I guess this issue expired (due to python2).

If you fully compiled this code, it would fail. CPython checks the well-formedness of __future__ statements after ast parsing.