Author meador.inge
Recipients meador.inge
Date 2011-09-09.03:11:05
SpamBayes Score 0.000212442
Marked as misclassified No
Message-id <1315537867.15.0.614423357455.issue12943@psf.upfronthosting.co.za>
In-reply-to
Content
In 2.x, 'python -m tokenize' worked great:

[meadori@motherbrain cpython]$ python2.7 -m tokenize test.py
1,0-1,5:        NAME    'print'
1,6-1,21:       STRING  '"Hello, World!"'
1,21-1,22:      NEWLINE '\n'
2,0-2,0:        ENDMARKER       ''

In 3.x, however, the functionality has been removed and replaced with
some hard-wired test code:

[meadori@motherbrain cpython]$ python3 -m tokenize test.py
TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3),
line='def parseline(self, line):')
TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1,
13), line='def parseline(self, line):')
TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14),
line='def parseline(self, line):')
...

The functionality was removed here [1], but with no explanation.  Let's add it back and document the functionality this time around.

[1] http://hg.python.org/cpython/rev/51e24512e305/
History
Date User Action Args
2011-09-09 03:11:07meador.ingesetrecipients: + meador.inge
2011-09-09 03:11:07meador.ingesetmessageid: <1315537867.15.0.614423357455.issue12943@psf.upfronthosting.co.za>
2011-09-09 03:11:06meador.ingelinkissue12943 messages
2011-09-09 03:11:05meador.ingecreate