This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: `ast.AST._attributes` is used by `ast.dump()` but not documented
Type: Stage:
Components: Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, brett.cannon, pekka.klarck, terry.reedy, yselivanov
Priority: normal Keywords:

Created on 2019-11-11 14:31 by pekka.klarck, last changed 2022-04-11 14:59 by admin.

Messages (7)
msg356362 - (view) Author: Pekka Klärck (pekka.klarck) Date: 2019-11-11 14:31
We have implemented an ast for our own tool so that we extend Python's standard `ast.AST`. When using `ast.dump(node, include_attributes=True)`, we were surprised to notice that line numbers weren't dumped although the docs of `ast.dump()` said they should be and we had set both `lineno` and `end_lineno` attributes to our nodes.

Looking at the implementation of `ast.dump()` revealed that `include_attributes=True` alone wasn't enough, the node also needed to have `_attributes` attribute similarly as it needs to have `_fields` for `ast.dump()` to be able to dump also child nodes. The problem is that `_attributes` isn't documented at all. Related to that, `ast.dump()` using `_fields` isn't documented either, but at least `_fields` is mentioned in the docs otherwise. Relevant links to `ast.dump()` docs and to the source code below.

`ast.AST` documents the `_fields` attribute but doesn't mention similar `_attributes`:
https://docs.python.org/3/library/ast.html#ast.AST

`ast.dump()` documents the `include_attributes` argument but doesn't mention that `_attributes` is needed (or that `_fields` needs to be set correctly to get child notes dumped):
https://docs.python.org/3/library/ast.html#ast.dump

`ast.dump()` source code shows that the undocumented `_attributes` needs to be set in addition to using `include_attributes=True`:
https://github.com/python/cpython/blob/3.8/Lib/ast.py#L123
msg356363 - (view) Author: Pekka Klärck (pekka.klarck) Date: 2019-11-11 14:36
Based on the `ast` source code there are also other functions that use  the undocumented `_attributes` attribute:

- copy_location
- fix_missing_locations
- increment_lineno
msg356713 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-15 20:05
A leading underscore usually means that the name is private and should not be relied on.  We of course use private names to implement public objects, else they should not exist.  '_fields' is an exception, and I leave it to the ast experts to decide if '_attributes' should be also.
msg356738 - (view) Author: Pekka Klärck (pekka.klarck) Date: 2019-11-16 00:15
I know attributes starting with an underscore are typically considered private, but the already mentioned `_fields` clearly isn't and, for example, `namedtuple` has several methods like `_asdict()` and `_replace()` that are documented to be part of the public API. `_attributes` clearly is part of the API as well but it isn't at all documented.
msg357027 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-20 04:15
Nosying the ast experts, which I meant to do before.
msg357086 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-11-20 17:15
It's also possible the leading underscore was used to prevent name collisions.

Unfortunately I don't know the history of _attributes, so without someone digging through the git history or personally knowing the reason behind it I don't know whether we should be exposing it.
msg357226 - (view) Author: Pekka Klärck (pekka.klarck) Date: 2019-11-21 22:23
I'd say `_attributes` is already exposed as defining it in your own node affects many of the functions in the ast module. For example, `ast.dump(node, include_attributes=True)` makes no sense otherwise.

Whatever was the reason for the leading underscore must be the same reason that added the underscore to `_fields`. That attribute is already documented.
History
Date User Action Args
2022-04-11 14:59:23adminsetgithub: 82946
2019-11-21 22:23:42pekka.klarcksetmessages: + msg357226
2019-11-20 17:15:04brett.cannonsetmessages: + msg357086
2019-11-20 04:15:48terry.reedysetnosy: + brett.cannon, benjamin.peterson, yselivanov
messages: + msg357027
2019-11-16 00:15:32pekka.klarcksetmessages: + msg356738
2019-11-15 20:05:39terry.reedysetnosy: + terry.reedy
messages: + msg356713
2019-11-11 14:36:05pekka.klarcksetmessages: + msg356363
2019-11-11 14:31:03pekka.klarckcreate