Title: Carry comments with the AST
Type: enhancement Stage: test needed
Components: Library (Lib) Versions: Python 3.5
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, Mark.Shannon, brett.cannon, christian.heimes, gvanrossum, kernc, louielu, mbdevpl, pitrou, rhettinger, serhiy.storchaka
Priority: low Keywords:

Created on 2015-05-03 15:21 by brett.cannon, last changed 2020-03-13 01:04 by gvanrossum.

Messages (15)
msg242485 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-03 15:21
One thing about is that it makes comments potentially semantically meaningful. Unfortunately the AST doesn't carry comments with it in any way, making it difficult to build a tool to implement a linter for PEP 484 using purely the ast module. Even if comments were carried along side-band and could do correlation by line number would be useful in this scenario.

I thought an issue had previously existed for this topic but I could find it.
msg242499 - (view) Author: Mark Shannon (Mark.Shannon) * (Python committer) Date: 2015-05-03 18:52
Comments don't belong on the AST. Where would you attach them?

The tokenizer module provides all information about comments. Tools can get the information quite easily if they need it.
msg242560 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-04 14:15
Normally I would agree comments don't belong there, but if we are going to start giving them semantic meaning then I don't think it's not so clear to me anymore.

As to where to attach, simple place is off of the Module node. Another is to have it be fundamental like lineno and only attach it when it is a line-trailing comment.

Yes, the tokenize module will give you the comments as well, but it is unfortunate you have to parse the code twice in order to get the comments and the AST.
msg242602 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-05-05 14:32
Another option is to provide a tool in 'tokenize' or 'ast' which will take the source and some comment regex and then attach the found comment metadata to the AST.
msg242603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-05-05 14:39
Or a separate AST node -> comment mapping.
msg289643 - (view) Author: Mateusz Bysiek (mbdevpl) Date: 2017-03-15 06:40
For some time now, there's an alternate ast implementation that carries PEP 484 type comments with the AST as attributes of certain nodes.

Their approach is described here:

If type comments become mainstream in Python, could this approach maybe be adopted as official Python AST at some point?
msg289685 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-03-15 18:21
The type annotation is already in the AST so there's nothing to carry over from typed_ast (we only care about the latest Python version while typed_ast tries to be version-agnostic).
msg292655 - (view) Author: Louie Lu (louielu) * Date: 2017-05-01 08:51
Brett, which implement method will you prefer?

If we want to carry comment at builtin_compile_impl, it will need to change the grammar since tokenize just drop the comment when dealing with source code.

But if just using regex, will it be more easy with just combine the exists tokenize and ast module?
msg292661 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-01 13:16
After PEP 526, the need for this proposal may have evaporated.
msg292682 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-05-01 17:50
There's potentially some usefulness from other tools, but Raymond is right that the main motivation is definitely gone long-term. Dropping this down to "low" priority simply because others have asked for this kind of support before.
msg292683 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-01 17:57
Where would comment be attached in the following case?

a = ('start'         # comment 1
     'continuation') # comment 2
msg333540 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-01-13 04:18
Since the AST carries the module/lineno attributes isn't there already a way trace back into the token stream to recover comments?
msg333541 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-01-13 04:19
Possible superceder:
msg334072 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-01-19 21:58
See also issue35766.
msg364061 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-03-13 01:04
I propose to close this issue, since (as of Python 3.8) we now have ast.parse(source, type_comments=True).
Date User Action Args
2020-03-13 01:04:47gvanrossumsetmessages: + msg364061
2020-03-12 20:11:39BTaskayasetnosy: + BTaskaya
2019-01-19 21:58:29gvanrossumsetnosy: + gvanrossum
messages: + msg334072
2019-01-13 04:19:17rhettingersetmessages: + msg333541
2019-01-13 04:18:40rhettingersetmessages: + msg333540
2019-01-12 09:19:41kerncsetnosy: + kernc
2017-05-01 17:57:59serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg292683
2017-05-01 17:50:33brett.cannonsetpriority: normal -> low

messages: + msg292682
2017-05-01 13:16:27rhettingersetnosy: + rhettinger
messages: + msg292661
2017-05-01 08:51:37louielusetnosy: + louielu
messages: + msg292655
2017-03-15 18:21:40brett.cannonsetmessages: + msg289685
2017-03-15 06:40:23mbdevplsetnosy: + mbdevpl
messages: + msg289643
2015-05-05 14:39:26pitrousetnosy: + pitrou
messages: + msg242603
2015-05-05 14:32:17brett.cannonsetmessages: + msg242602
2015-05-04 14:33:51christian.heimessetnosy: + christian.heimes
2015-05-04 14:15:13brett.cannonsetmessages: + msg242560
2015-05-03 18:52:45Mark.Shannonsetnosy: + Mark.Shannon
messages: + msg242499
2015-05-03 15:21:10brett.cannoncreate