This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author pablogsal
Recipients Andrew.C, Anthony Sottile, Jim Fasarakis-Hilliard, amaury.forgeotdarc, berker.peksag, djmitche, effbot, kirkshorts, meador.inge, pablogsal, serhiy.storchaka, superluser
Date 2021-01-27.20:58:53
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1611781133.91.0.455503481075.issue3353@roundup.psfhosted.org>
In-reply-to
Content
I have built a draft of how the changes required to make what you describe, in case you want to finish them:

https://github.com/pablogsal/cpython/tree/tokenizer_mod
History
Date User Action Args
2021-01-27 20:58:53pablogsalsetrecipients: + pablogsal, effbot, amaury.forgeotdarc, djmitche, kirkshorts, meador.inge, berker.peksag, serhiy.storchaka, superluser, Andrew.C, Anthony Sottile, Jim Fasarakis-Hilliard
2021-01-27 20:58:53pablogsalsetmessageid: <1611781133.91.0.455503481075.issue3353@roundup.psfhosted.org>
2021-01-27 20:58:53pablogsallinkissue3353 messages
2021-01-27 20:58:53pablogsalcreate