classification
Title: tokenize: add python -m tokenize support back
Type: enhancement Stage: resolved
Components: Documentation, Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: meador.inge Nosy List: docs@python, eric.araujo, georg.brandl, meador.inge, python-dev
Priority: normal Keywords: needs review, patch

Created on 2011-09-09 03:11 by meador.inge, last changed 2011-10-07 13:56 by meador.inge. This issue is now closed.

Files
File name Uploaded Description Edit
issue12943.patch meador.inge, 2011-09-11 04:33 Patch against tip (3.3.0a0) review
issue12943-2.patch meador.inge, 2011-09-17 17:41 v2 patch against tip (3.3.0a0) review
issue12943-3.patch meador.inge, 2011-09-19 13:31 v3 patch against tip (3.3.0a0) review
issue12943-4.patch meador.inge, 2011-09-20 03:34 v4 patch against tip (3.3.0a0) review
issue12943-5.patch meador.inge, 2011-09-23 03:55 v5 patch against tip (3.3.0a0) review
issue12943-6.patch meador.inge, 2011-10-03 02:09 v6 patch against tip (3.3.0a0) review
Messages (14)
msg143752 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-09 03:11
In 2.x, 'python -m tokenize' worked great:

[meadori@motherbrain cpython]$ python2.7 -m tokenize test.py
1,0-1,5:        NAME    'print'
1,6-1,21:       STRING  '"Hello, World!"'
1,21-1,22:      NEWLINE '\n'
2,0-2,0:        ENDMARKER       ''

In 3.x, however, the functionality has been removed and replaced with
some hard-wired test code:

[meadori@motherbrain cpython]$ python3 -m tokenize test.py
TokenInfo(type=57 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='')
TokenInfo(type=1 (NAME), string='def', start=(1, 0), end=(1, 3),
line='def parseline(self, line):')
TokenInfo(type=1 (NAME), string='parseline', start=(1, 4), end=(1,
13), line='def parseline(self, line):')
TokenInfo(type=53 (OP), string='(', start=(1, 13), end=(1, 14),
line='def parseline(self, line):')
...

The functionality was removed here [1], but with no explanation.  Let's add it back and document the functionality this time around.

[1] http://hg.python.org/cpython/rev/51e24512e305/
msg143853 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-11 04:33
Patch against tip.
msg144210 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-17 17:41
v2 patch which addresses comments made by merwok via rietveld.
msg144211 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-09-17 17:45
Looks good to me. Did you get commit rights already?
msg144213 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-17 18:00
> Did you get commit rights already?

I have not.  I still need to submit a contributor agreement as well.  I
plan to fax that today.
msg144268 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-19 13:31
Updated patch fixing some issues pointed out by Ezio on Rietveld.
msg144317 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-20 03:34
Updated patch which adds and documents a '-h,--help' option.  This option
was suggested during code review.
msg144405 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-09-22 08:13
I don't think the help option needs to be documented, it will document itself.

An additional suggestion is to catch errors on tokenizing and exiting with a non-zero exit status in these cases.
msg144435 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-09-23 03:55
> I don't think the help option needs to be documented, it will document 
> itself.

Normally I would document it anyway, but in this case there is only the
one option.  So, I dropped it.

> An additional suggestion is to catch errors on tokenizing and exiting 
> with a non-zero exit status in these cases.

I like that idea.  Sample output:

$ python -m tokenize foo.py
foo.py: error: unknown encoding: meador
$ python -m tokenize foo.py
foo.py:7:2: error: unindent does not match any outer indentation level
$ ./python -m tokenize bogus.py
error: [Errno 2] No such file or directory: 'bogus.py'
$ python -m tokenize foo.py
$ echo '[1,' | ./python -m tokenize
1,0-1,1:            OP             '['            
1,1-1,2:            NUMBER         '1'            
1,2-1,3:            OP             ','            
1,3-1,4:            NL             '\n'           
<stdin>:2:0: error: EOF in multi-line statement

Attached is the latest patch implementing these suggestions.
msg144437 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2011-09-23 05:20
I would say this looks good now.
msg144800 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-03 02:09
Fixed a few more nits pointed out in review.
msg145003 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-06 12:44
I made a few last remarks on Rietveld; feel free to address or ignore them and commit right away.
msg145074 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2011-10-07 13:51
Éric, thanks.  I fixed most of your points.  And thanks to everyone that reviewed this.  Much appreciated.
msg145075 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-10-07 13:55
New changeset 85254bb6c1c6 by Meador Inge in branch 'default':
Issue #12943: python -m tokenize support has been added to tokenize.
http://hg.python.org/cpython/rev/85254bb6c1c6
History
Date User Action Args
2011-10-07 13:56:53meador.ingesetstatus: open -> closed
assignee: docs@python -> meador.inge
resolution: fixed
stage: patch review -> resolved
2011-10-07 13:55:04python-devsetnosy: + python-dev
messages: + msg145075
2011-10-07 13:51:56meador.ingesetmessages: + msg145074
2011-10-06 12:44:43eric.araujosetnosy: + eric.araujo
messages: + msg145003
2011-10-03 02:09:08meador.ingesetfiles: + issue12943-6.patch

messages: + msg144800
2011-09-23 05:20:10georg.brandlsetmessages: + msg144437
2011-09-23 03:55:25meador.ingesetfiles: + issue12943-5.patch

messages: + msg144435
2011-09-22 08:13:21georg.brandlsetmessages: + msg144405
2011-09-20 03:34:01meador.ingesetfiles: + issue12943-4.patch

messages: + msg144317
2011-09-19 13:31:06meador.ingesetfiles: + issue12943-3.patch

messages: + msg144268
2011-09-17 18:00:20meador.ingesetmessages: + msg144213
2011-09-17 17:45:54georg.brandlsetnosy: + georg.brandl
messages: + msg144211
2011-09-17 17:41:24meador.ingesetfiles: + issue12943-2.patch

messages: + msg144210
2011-09-11 04:33:50meador.ingesetfiles: + issue12943.patch

assignee: docs@python
components: + Documentation

keywords: + patch, needs review
nosy: + docs@python
messages: + msg143853
stage: needs patch -> patch review
2011-09-09 03:11:06meador.ingecreate