This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author takluyver
Recipients Devin Jeanpierre, eric.araujo, eric.snow, martin.panter, meador.inge, petri.lehtinen, serhiy.storchaka, takluyver, terry.reedy, vstinner
Date 2018-03-11.09:22:54
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
> Why not just bless the existing generate_tokens() function as a public API

We're actually using generate_tokens() from IPython - we wanted a way to tokenize unicode strings, and although it's undocumented, it's been there for a number of releases and does what we want. So +1 to promoting it to a public API.

In fact, at the moment, IPython has its own copy of tokenize to fix one or two old issues. I'm trying to get rid of that and use the stdlib module again, which is how I came to notice that we're using an undocumented API.
Date User Action Args
2018-03-11 09:22:55takluyversetrecipients: + takluyver, terry.reedy, vstinner, Devin Jeanpierre, eric.araujo, meador.inge, eric.snow, petri.lehtinen, martin.panter, serhiy.storchaka
2018-03-11 09:22:55takluyversetmessageid: <>
2018-03-11 09:22:55takluyverlinkissue12486 messages
2018-03-11 09:22:54takluyvercreate