This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author djmitche
Recipients Andrew.C, amaury.forgeotdarc, djmitche, effbot, kirkshorts, meador.inge
Date 2015-04-14.13:35:07
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1429018508.04.0.223784779841.issue3353@psf.upfronthosting.co.za>
In-reply-to
Content
From my read of this bug, there are two distinct tasks mentioned:

 1. make PyTokenizer_* part of the Python-level API
 2. re-implement 'tokenize' in terms of that Python-level API

#1 is largely complete in Andrew's latest patch, but that will likely need:
 * rebasing
 * hiding struct fields
 * documentation

#2 is, I think, a separate project.  There may be good reasons *not* to do this which I'm not aware of, and barring such reasons the rewrite will be difficult and could potentially change behavior like issue2180.  So I would suggest filing a new issue for #2 when #1 is complete.  And I'll work on #1.
History
Date User Action Args
2015-04-14 13:35:08djmitchesetrecipients: + djmitche, effbot, amaury.forgeotdarc, kirkshorts, meador.inge, Andrew.C
2015-04-14 13:35:08djmitchesetmessageid: <1429018508.04.0.223784779841.issue3353@psf.upfronthosting.co.za>
2015-04-14 13:35:08djmitchelinkissue3353 messages
2015-04-14 13:35:07djmitchecreate