This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author fpom
Recipients fpom, r.david.murray
Date 2017-10-08.15:46:01
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1507477561.47.0.213398074469.issue31726@psf.upfronthosting.co.za>
In-reply-to
Content
I have the problem still exists in 3.4.3 at least:

Python 3.4.3 (default, Nov 17 2016, 01:08:31) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import token
>>> token.COMMENT
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'COMMENT'

About the use case: I often use tokenize to build recursive descendant parsers for domain-specific languages. Not having all the token constants can be worked around with code below, but it would be so better to have a complete module...  :)

import token
_tok = next(tokenize.tokenize(io.BytesIO(b"").readline))
token.tok_name[_tok.type] = "BACKQUOTE"
for number, name in token.tok_name.items() :
    if not hasattr(token, name) :
        setattr(token, name, number)
History
Date User Action Args
2017-10-08 15:46:01fpomsetrecipients: + fpom, r.david.murray
2017-10-08 15:46:01fpomsetmessageid: <1507477561.47.0.213398074469.issue31726@psf.upfronthosting.co.za>
2017-10-08 15:46:01fpomlinkissue31726 messages
2017-10-08 15:46:01fpomcreate