the token module defines constants for token types e.g

 NAME = 1
 NUMBER = 2
 STRING = 3
 NEWLINE = 4

etc.

These constants are very useful for any code which needs to tokenize python source, yet they are not listed in the documentation.


Is this a documentation bug?

It is. Constants should be documented in the regular doc, not only by introspection or reading the source. See the doc for tokenize: “All constants from the token module are also exported from tokenize, as are two additional token type values”.

Thanks for the report. Would you mind writing a patch?

I'm attaching a documentation patch.

Do note that there is also a bit of code-level inconsistency: a few tokens (COMMENT, NL) are defined in tokenize module which is strange and inconvenient.

Should that be fixed too? (by moving token definitions into token module)

Five comments:

* I would list them all in one directive, like this:

.. data:: FOO
          BAR
          BAZ

  which makes the display more compact.

* There is no description in that directive.  Best move part of the
  description above them in it.

* NL and COMMENT are defined in tokenize because they are neither 
  defined nor used by the Python tokenizer.

> * I would list them all in one directive, like this:

Ok, done. Attaching the updated patch


> There is no description in that directive.  Best move part of the
description above them in it.

I am not sure I understand your request. Could you clarify?


>  NL and COMMENT are defined in tokenize because they are neither 
defined nor used by the Python tokenizer.

Oh, I just realized the source of my misunderstanding: token.py captures token types as they are returned by python's own tokenizer. While tokenize module  does its own tokenization and produces results which are a bit different. COMMENT and NL tokens is one of the differences but there is a difference in how the operation tokens are treated. In fact tokenize's docstring explicitly says so.

... It is designed to match the working of the Python tokenizer exactly, except that it produces COMMENT tokens for comments and gives type OP for all operators ...

I think this clarification should go into official docs as well, would you agree?

PS. even with this clarification, I find the situation quite a bit confusing: especially given that tok_name dict from token module does have a name for COMMENT and that name is inserted by tokenize module..  So I still feel that just adding COMMENT and NL to token module (with clarification that they are not generated by the python's own tokenizer) would be a bit cleaner.

Committed in r85614.