This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author gvanrossum
Recipients benjamin.peterson, gregory.p.smith, gvanrossum, lukasz.langa, serhiy.storchaka
Date 2018-04-23.04:48:05
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <>
I'm glad you've rediscovered pgen2!

I'm in favor of unifying the tokenizers and of updating and moving pgen2 (though I don't have time to do the work).

I'm not sure if it's technically possible to give the ability to tokenize Python 2.7 and up without some version-selection flag -- have you researched this part yet?

Also I think you may have to make a distinction between the parser generator and its data structures, and between the generated parser for Python vs. the parser for other LL(1) grammars one might feed into it.

And I don't think you're proposing to replace Parser/pgen.c with Lib/pgen/, right? Nor to replace the CST actually used by CPython's parser with the data structures used by pgen2's driver. So the relationship between the CST you propose to document and CPython internals wouldn't be quite the same as that between the AST used by CPython and the ast module (since those *do* actually use the same code).
Date User Action Args
2018-04-23 04:48:05gvanrossumsetrecipients: + gvanrossum, gregory.p.smith, benjamin.peterson, lukasz.langa, serhiy.storchaka
2018-04-23 04:48:05gvanrossumsetmessageid: <>
2018-04-23 04:48:05gvanrossumlinkissue33337 messages
2018-04-23 04:48:05gvanrossumcreate