➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author mbussonn
Recipients Devin Jeanpierre, eric.araujo, eric.snow, martin.panter, mbussonn, meador.inge, petri.lehtinen, serhiy.storchaka, takluyver, terry.reedy, vstinner
Date 2018-05-17.20:28:39
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1526588919.28.0.682650639539.issue12486@psf.upfronthosting.co.za>
In-reply-to
Content 
> Why not just bless the existing generate_tokens() function as a public API, 

Yes please, or just make the private `_tokenize` public under another name. The `tokenize.tokenize` method try to magically detect encoding which may be unnecessary.
History
Date User Action Args
2018-05-17 20:28:39mbussonnsetrecipients: + mbussonn, terry.reedy, vstinner, Devin Jeanpierre, eric.araujo, meador.inge, eric.snow, takluyver, petri.lehtinen, martin.panter, serhiy.storchaka
2018-05-17 20:28:39mbussonnsetmessageid: <1526588919.28.0.682650639539.issue12486@psf.upfronthosting.co.za>
2018-05-17 20:28:39mbussonnlinkissue12486 messages
2018-05-17 20:28:39mbussonncreate