I'm glad you've rediscovered pgen2!

I'm in favor of unifying the tokenizers and of updating and moving pgen2 (though I don't have time to do the work).

I'm not sure if it's technically possible to give tokenize.py the ability to tokenize Python 2.7 and up without some version-selection flag -- have you researched this part yet?

Also I think you may have to make a distinction between the parser generator and its data structures, and between the generated parser for Python vs. the parser for other LL(1) grammars one might feed into it.

And I don't think you're proposing to replace Parser/pgen.c with Lib/pgen/, right? Nor to replace the CST actually used by CPython's parser with the data structures used by pgen2's driver. So the relationship between the CST you propose to document and CPython internals wouldn't be quite the same as that between the AST used by CPython and the ast module (since those *do* actually use the same code).