In 2.x, 'python -m tokenize' worked great:

[meadori@motherbrain cpython]$ python2.7 -m tokenize test.py
1,0-1,5:        NAME    'print'
1,6-1,21:       STRING  '"Hello, World!"'
1,21-1,22:      NEWLINE '\n'
2,0-2,0:        ENDMARKER       ''

In 3.x, however, the functionality has been removed and replaced with
some hard-wired test code:

[meadori@motherbrain cpython]$ python3 -m tokenize test.py
TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3),
line='def parseline(self, line):')
TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1,
13), line='def parseline(self, line):')
TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14),
line='def parseline(self, line):')
...

The functionality was removed here [1], but with no explanation.  Let's add it back and document the functionality this time around.

[1] http://hg.python.org/cpython/rev/51e24512e305/

Patch against tip.

v2 patch which addresses comments made by merwok via rietveld.

Looks good to me. Did you get commit rights already?

> Did you get commit rights already?

I have not.  I still need to submit a contributor agreement as well.  I
plan to fax that today.

Updated patch fixing some issues pointed out by Ezio on Rietveld.

Updated patch which adds and documents a '-h,--help' option.  This option
was suggested during code review.

I don't think the help option needs to be documented, it will document itself.

An additional suggestion is to catch errors on tokenizing and exiting with a non-zero exit status in these cases.

> I don't think the help option needs to be documented, it will document 
> itself.

Normally I would document it anyway, but in this case there is only the
one option.  So, I dropped it.

> An additional suggestion is to catch errors on tokenizing and exiting 
> with a non-zero exit status in these cases.

I like that idea.  Sample output:

$ python -m tokenize foo.py
foo.py: error: unknown encoding: meador
$ python -m tokenize foo.py
foo.py:7:2: error: unindent does not match any outer indentation level
$ ./python -m tokenize bogus.py
error: [Errno 2] No such file or directory: 'bogus.py'
$ python -m tokenize foo.py
$ echo '[1,' | ./python -m tokenize
1,0-1,1:            OP             '['            
1,1-1,2:            NUMBER         '1'            
1,2-1,3:            OP             ','            
1,3-1,4:            NL             '\n'           
<stdin>:2:0: error: EOF in multi-line statement

Attached is the latest patch implementing these suggestions.

I would say this looks good now.

Fixed a few more nits pointed out in review.

I made a few last remarks on Rietveld; feel free to address or ignore them and commit right away.

Éric, thanks.  I fixed most of your points.  And thanks to everyone that reviewed this.  Much appreciated.

New changeset 85254bb6c1c6 by Meador Inge in branch 'default':
Issue #12943: python -m tokenize support has been added to tokenize.
http://hg.python.org/cpython/rev/85254bb6c1c6