classification
Title: Carry comments with the AST
Type: enhancement Stage: test needed
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Mark.Shannon, brett.cannon, christian.heimes, mbdevpl, pitrou
Priority: normal Keywords:

Created on 2015-05-03 15:21 by brett.cannon, last changed 2017-03-15 18:21 by brett.cannon.

Messages (7)
msg242485 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-03 15:21
One thing about https://www.python.org/dev/peps/pep-0484/ is that it makes comments potentially semantically meaningful. Unfortunately the AST doesn't carry comments with it in any way, making it difficult to build a tool to implement a linter for PEP 484 using purely the ast module. Even if comments were carried along side-band and could do correlation by line number would be useful in this scenario.

I thought an issue had previously existed for this topic but I could find it.
msg242499 - (view) Author: Mark Shannon (Mark.Shannon) * Date: 2015-05-03 18:52
Comments don't belong on the AST. Where would you attach them?

The tokenizer module provides all information about comments. Tools can get the information quite easily if they need it.
msg242560 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-04 14:15
Normally I would agree comments don't belong there, but if we are going to start giving them semantic meaning then I don't think it's not so clear to me anymore.

As to where to attach, simple place is off of the Module node. Another is to have it be fundamental like lineno and only attach it when it is a line-trailing comment.

Yes, the tokenize module will give you the comments as well, but it is unfortunate you have to parse the code twice in order to get the comments and the AST.
msg242602 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-05 14:32
Another option is to provide a tool in 'tokenize' or 'ast' which will take the source and some comment regex and then attach the found comment metadata to the AST.
msg242603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-05-05 14:39
Or a separate AST node -> comment mapping.
msg289643 - (view) Author: Mateusz Bysiek (mbdevpl) Date: 2017-03-15 06:40
For some time now, there's an alternate ast implementation https://github.com/python/typed_ast that carries PEP 484 type comments with the AST as attributes of certain nodes.

Their approach is described here: https://github.com/python/typed_ast/blob/master/typed_ast/ast3.py#L5

If type comments become mainstream in Python, could this approach maybe be adopted as official Python AST at some point?
msg289685 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-15 18:21
The type annotation is already in the AST so there's nothing to carry over from typed_ast (we only care about the latest Python version while typed_ast tries to be version-agnostic).
History
Date User Action Args
2017-03-15 18:21:40brett.cannonsetmessages: + msg289685
2017-03-15 06:40:23mbdevplsetnosy: + mbdevpl
messages: + msg289643
2015-05-05 14:39:26pitrousetnosy: + pitrou
messages: + msg242603
2015-05-05 14:32:17brett.cannonsetmessages: + msg242602
2015-05-04 14:33:51christian.heimessetnosy: + christian.heimes
2015-05-04 14:15:13brett.cannonsetmessages: + msg242560
2015-05-03 18:52:45Mark.Shannonsetnosy: + Mark.Shannon
messages: + msg242499
2015-05-03 15:21:10brett.cannoncreate