I wrote an AST optimizer which build AST from source code using ast.parse(), modify AST in place, and then compile the AST to bytecode.

The problem is that the lineno and col_offset attributes of the "arg" AST node are uninitialized when AST is rebuild from Python objects: compile(ast_tree, ...). As a consequence, the compilation may fail because lineno or col_offset values are invalid.

Attached patch enhances Parser/asdl_c.py to take "arg" attributes in account. It also updates the generated files Include/Python-ast.h and Python/Python-ast.c, and fix usage of the arg() function in Python/ast.c.

It looks like only the "arg" AST node had attributes which were initialized (other AST nodes are handled with a different code path which already filled attributes).

See Parser/Python.asdl for the definition of the "arg" AST node.

Note: Python 2.7 is not affected. In Python 2, function arguments are simple expressions, they don't have a dedicated type with lineno and col_offset attributes. "arg" was introduced in Python 3 with the PEP-3107 (function annotations).

Hi Victor,

1. Result of the compilation -> success
2. I have checked your patch, I think there is no problem.
3. How can I test it? currently, everything is fine.

For me, this patch is ok and working, but I need the point 3.

New changeset 54d4290f0ec6 by Victor Stinner in branch 'default':
Issue bpo-25555: Fix parser and AST: fill lineno and col_offset of "arg" node when
https://hg.python.org/cpython/rev/54d4290f0ec6

3. How can I test it? currently, everything is fine.

Sorry, I don't know how to test it.

To write my fix, I had to modify the compile() builtin function to be able to compile Python to AST, and then AST to Python. But without this fix I don't know how to test.

I tried to revert the AST patch on my FAT Python to try to find which part of the code fails without the fix, but FAT Python doesn't compile anymore. I reverted the revert, and it still doesn't work!?

For this bug, sorry, you have to trust me :-(

I applied the fix to Python 3.6. I will wait a bit for buildbots before backporting to 3.5.

I trust you, I have reviewed the source code and it seems to be right.

New changeset 2d8988f64cea by Victor Stinner in branch '3.5':
Issue bpo-25555: Fix parser and AST: fill lineno and col_offset of "arg" node when
https://hg.python.org/cpython/rev/2d8988f64cea

I backported the fix to Python 3.5. Since I'm not convinced that the bug can be triggered without modifying CPython (the bug only impacts FAT Python which installs a hook on the AST compiler), I don't want to backport it to Python 2.7 to avoid the risk of regression.