classification
Title: Fix parser and AST: fill lineno and col_offset when compiling AST from Python objects
Type: Stage:
Components: Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, matrixise, python-dev, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2015-11-05 09:42 by vstinner, last changed 2015-11-09 21:14 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
ast.patch vstinner, 2015-11-05 09:42 review
Messages (7)
msg254093 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-05 09:42
I wrote an AST optimizer which build AST from source code using ast.parse(), modify AST in place, and then compile the AST to bytecode.

The problem is that the lineno and col_offset attributes of the "arg" AST node are uninitialized when AST is rebuild from Python objects: compile(ast_tree, ...). As a consequence, the compilation may fail because lineno or col_offset values are invalid.

Attached patch enhances Parser/asdl_c.py to take "arg" attributes in account. It also updates the generated files Include/Python-ast.h and Python/Python-ast.c, and fix usage of the arg() function in Python/ast.c.

It looks like only the "arg" AST node had attributes which were initialized (other AST nodes are handled with a different code path which already filled attributes).

See Parser/Python.asdl for the definition of the "arg" AST node.

Note: Python 2.7 is not affected. In Python 2, function arguments are simple expressions, they don't have a dedicated type with lineno and col_offset attributes. "arg" was introduced in Python 3 with the PEP 3107 (function annotations).
msg254155 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2015-11-06 01:07
Hi Victor,

1. Result of the compilation -> success
2. I have checked your patch, I think there is no problem.
3. How can I test it? currently, everything is fine.

For me, this patch is ok and working, but I need the point 3.
msg254202 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-06 16:02
New changeset 54d4290f0ec6 by Victor Stinner in branch 'default':
Issue #25555: Fix parser and AST: fill lineno and col_offset of "arg" node when
https://hg.python.org/cpython/rev/54d4290f0ec6
msg254203 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-06 16:05
> 3. How can I test it? currently, everything is fine.

Sorry, I don't know how to test it.

To write my fix, I had to modify the compile() builtin function to be able to compile Python to AST, and then AST to Python. But without this fix I don't know how to test.

I tried to revert the AST patch on my FAT Python to try to find which part of the code fails without the fix, but FAT Python doesn't compile anymore. I reverted the revert, and it still doesn't work!?

For this bug, sorry, you have to trust me :-(

I applied the fix to Python 3.6. I will wait a bit for buildbots before backporting to 3.5.
msg254204 - (view) Author: Stéphane Wirtel (matrixise) * (Python committer) Date: 2015-11-06 16:07
I trust you, I have reviewed the source code and it seems to be right.
msg254403 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-09 20:56
New changeset 2d8988f64cea by Victor Stinner in branch '3.5':
Issue #25555: Fix parser and AST: fill lineno and col_offset of "arg" node when
https://hg.python.org/cpython/rev/2d8988f64cea
msg254409 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-09 21:14
I backported the fix to Python 3.5. Since I'm not convinced that the bug can be triggered without modifying CPython (the bug only impacts FAT Python which installs a hook on the AST compiler), I don't want to backport it to Python 2.7 to avoid the risk of regression.
History
Date User Action Args
2015-11-09 21:14:10vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg254409
2015-11-09 20:56:15python-devsetmessages: + msg254403
2015-11-06 16:07:51matrixisesetmessages: + msg254204
2015-11-06 16:05:58vstinnersetmessages: + msg254203
2015-11-06 16:02:35python-devsetnosy: + python-dev
messages: + msg254202
2015-11-06 01:07:21matrixisesetmessages: + msg254155
2015-11-05 09:42:04vstinnercreate