diff --git a/Doc/library/tokenize.rst b/Doc/library/tokenize.rst --- a/Doc/library/tokenize.rst +++ b/Doc/library/tokenize.rst @@ -15,6 +15,12 @@ implemented in Python. The scanner in t well, making it useful for implementing "pretty-printers," including colorizers for on-screen displays. +To simplify token stream handling, all :ref:`operators` and :ref:`delimiters` +tokens are returned using the generic :data:`token.OP` token type. The exact +type can be determined by checking the token ``string`` field on the +:term:`named tuple` returned from :func:`tokenize.tokenize` for the character +sequence that identifies a specific operator token. + The primary entry point is a :term:`generator`: .. function:: generate_tokens(readline)