This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gpolo
Recipients gpolo
Date 2008-02-17.23:00:28
SpamBayes Score 0.02092547
Marked as misclassified No
Message-id <1203289230.78.0.790200923315.issue2134@psf.upfronthosting.co.za>
In-reply-to
Content
function generate_tokes at tokenize.py yields token OP (51) for colon,
while it should be token COLON (11). It probably affects other python
versions as well.

I'm attaching a minor sample that demonstrates this, running it returns
the following output:

1 'if' (1, 0) (1, 2) if a == 2:
1 'a' (1, 3) (1, 4) if a == 2:
51 '==' (1, 5) (1, 7) if a == 2:
2 '2' (1, 8) (1, 9) if a == 2:
51 ':' (1, 9) (1, 10) if a == 2:
1 'print' (2, 0) (2, 5) print 'hey'
3 "'hey'" (2, 6) (2, 11) print 'hey'
0 '' (3, 0) (3, 0)

I didn't check if there are problems with other tokens, I noticed this
with colon because I was trying to make some improvements on tabnanny.
History
Date User Action Args
2008-02-17 23:00:31gpolosetspambayes_score: 0.0209255 -> 0.02092547
recipients: + gpolo
2008-02-17 23:00:30gpolosetspambayes_score: 0.0209255 -> 0.0209255
messageid: <1203289230.78.0.790200923315.issue2134@psf.upfronthosting.co.za>
2008-02-17 23:00:29gpololinkissue2134 messages
2008-02-17 23:00:28gpolocreate