classification
Title: ast.NodeTransformer doc bug
Type: behavior Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: aronacher, benjamin.peterson, daishiharada, docs@python, georg.brandl, miss-islington, pablogsal, terry.reedy
Priority: normal Keywords: patch

Created on 2008-08-09 02:26 by daishiharada, last changed 2020-01-12 20:44 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17172 merged BTaskaya, 2019-11-15 15:14
PR 17972 merged miss-islington, 2020-01-12 20:39
PR 17973 merged miss-islington, 2020-01-12 20:39
Messages (7)
msg70923 - (view) Author: daishi (daishiharada) Date: 2008-08-09 02:26
I am testing python 2.6 from SVN version: 40110

I tried the following, based on the documentation
and example in the ast module. I would expect the
second 'compile' to succeed also, instead of
throwing an exception.

Python 2.6b2+ (unknown, Aug  6 2008, 18:05:08) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> a = ast.parse('foo', mode='eval')
>>> x = compile(a, '<unknown>', mode='eval')
>>> class RewriteName(ast.NodeTransformer):
...     def visit_Name(self, node):
...         return ast.copy_location(ast.Subscript(
...             value=ast.Name(id='data', ctx=ast.Load()),
...             slice=ast.Index(value=ast.Str(s=node.id)),
...             ctx=node.ctx
...         ), node)
... 
>>> a2 = RewriteName().visit(a)
>>> x2 = compile(a2, '<unknown>', mode='eval')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: required field "lineno" missing from expr
>>>
msg71209 - (view) Author: Armin Ronacher (aronacher) * (Python committer) Date: 2008-08-16 12:21
This is actually not a bug.  copy_location does not work recursively. 
For this example it's more useful to use the "fix_missing_locations"
function which traverses the tree and copies the locations from the
parent node to the child nodes:

import ast
a = ast.parse('foo', mode='eval')
x = compile(a, '<unknown>', mode='eval')

class RewriteName(ast.NodeTransformer):
    def visit_Name(self, node):
        return ast.Subscript(
            value=ast.Name(id='data', ctx=ast.Load()),
            slice=ast.Index(value=ast.Str(s=node.id)),
            ctx=node.ctx
        )

a2 = ast.fix_missing_locations(RewriteName().visit(a))
msg245968 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-06-29 20:44
I am reopening this as a doc bug because RewriteName is a copy (with 'ast.' prefixes added) of a buggy example in the doc. The bug is that the new .value Name and Str attributes do not get the required 'lineno' and 'col_offset' attributes.  As Armin said, copy_location is not recursive and does not fix children of the node it fixes.  Also, the recursive .visit method does not recurse into children of replacement nodes (and if it did, the new Str node would still not be fixed).

The fix could be to reuse the Name node and add another copy_location call: the following works.

    def visit_Name(self, node):
        return ast.copy_location(
            ast.Subscript(
                value=node,
                slice=ast.Index(value=ast.copy_location(
                    ast.Str(s=node.id), node)),
                ctx=node.ctx),
            node)

but I think this illustrates that comment in the fix_missing_locations() entry that locations are "tedious to fill in for generated nodes".  So I think the doc fix should use Armin's version of RewriteName and say to call fix_missing_locations on the result of .visit if new nodes are added.  (I checked that his code still works in 3.5).

The entry for NodeTransformer might mention that .visit does not recurse into replacement nodes.

The missing lineno error came up in this python-list thread:
https://mail.python.org/pipermail/python-list/2015-June/693316.html
msg356786 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-17 02:09
I re-verified the problem, its presence in the doc, and the fix with 3.9.
msg359872 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-01-12 20:38
New changeset 6680f4a9f5d15ab82b2ab6266c6f917cb78c919a by Pablo Galindo (Batuhan Ta┼čkaya) in branch 'master':
bpo-3530: Add advice on when to correctly use fix_missing_locations in the AST docs (GH-17172)
https://github.com/python/cpython/commit/6680f4a9f5d15ab82b2ab6266c6f917cb78c919a
msg359874 - (view) Author: miss-islington (miss-islington) Date: 2020-01-12 20:44
New changeset e222b4c69f99953a14ded52497a9909e34fc3893 by Miss Islington (bot) in branch '3.7':
bpo-3530: Add advice on when to correctly use fix_missing_locations in the AST docs (GH-17172)
https://github.com/python/cpython/commit/e222b4c69f99953a14ded52497a9909e34fc3893
msg359875 - (view) Author: miss-islington (miss-islington) Date: 2020-01-12 20:44
New changeset ef0af30e507a29dae03aae40459b9c44c96f260d by Miss Islington (bot) in branch '3.8':
bpo-3530: Add advice on when to correctly use fix_missing_locations in the AST docs (GH-17172)
https://github.com/python/cpython/commit/ef0af30e507a29dae03aae40459b9c44c96f260d
History
Date User Action Args
2020-01-12 20:44:37miss-islingtonsetmessages: + msg359875
2020-01-12 20:44:29miss-islingtonsetnosy: + miss-islington
messages: + msg359874
2020-01-12 20:40:00pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-01-12 20:39:11miss-islingtonsetpull_requests: + pull_request17380
2020-01-12 20:39:04miss-islingtonsetpull_requests: + pull_request17379
2020-01-12 20:38:56pablogsalsetnosy: + pablogsal
messages: + msg359872
2019-11-17 02:10:56terry.reedysetversions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.4, Python 3.5, Python 3.6
2019-11-17 02:09:54terry.reedysetmessages: + msg356786
2019-11-15 15:14:43BTaskayasetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request16681
2015-06-29 20:44:43terry.reedysetstatus: closed -> open

resolution: not a bug -> (no value)

assignee: docs@python
stage: needs patch
title: ast.NodeTransformer bug -> ast.NodeTransformer doc bug
nosy: + terry.reedy, docs@python, benjamin.peterson
versions: + Python 2.7, Python 3.4, Python 3.5, Python 3.6, - Python 2.6
messages: + msg245968
components: + Documentation
type: behavior
2008-08-16 16:31:07benjamin.petersonsetstatus: open -> closed
resolution: not a bug
2008-08-16 12:21:28aronachersetmessages: + msg71209
2008-08-11 18:02:05georg.brandlsetnosy: + georg.brandl, aronacher
2008-08-09 02:26:13daishiharadacreate