> * I would list them all in one directive, like this:

Ok, done. Attaching the updated patch


> There is no description in that directive.  Best move part of the
description above them in it.

I am not sure I understand your request. Could you clarify?


>  NL and COMMENT are defined in tokenize because they are neither 
defined nor used by the Python tokenizer.

Oh, I just realized the source of my misunderstanding: token.py captures token types as they are returned by python's own tokenizer. While tokenize module  does its own tokenization and produces results which are a bit different. COMMENT and NL tokens is one of the differences but there is a difference in how the operation tokens are treated. In fact tokenize's docstring explicitly says so.

... It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators ...

I think this clarification should go into official docs as well, would you agree?

PS. even with this clarification, I find the situation quite a bit confusing: especially given that tok_name dict from token module does have a name for COMMENT and that name is inserted by tokenize module..  So I still feel that just adding COMMENT and NL to token module (with clarification that they are not generated by the python's own tokenizer) would be a bit cleaner.