+1 in general to this work. Łukasz is effectively preaching to the choir by looping me in here. :)

It is a big challenge to practically support Python in that we have no good ability to parse and understand all language syntax versions via a single API that does not depend on the version of the language your own tools process is running under.

lib2to3.pgen2 is the closest thing we've got and it used by a notable crop of python refactoring tools today because there really wasn't another available choice.  All they know is that they've got a ".py" file, they can't know which specific language versions it may be intended for.  Nor should they ever need to run _on_ that language version.  That situation is a nightmare (ex: pylint uses ast and must run on the version of the language it is to analyze as)

I'd love to see a ponycorn module that everything could use to run on top of Python 3.recent yet be able to meaningfully process 2.7 and 3.4-3.7 code.  This is an area where the language versions we support parsing and analyzing should _not_ be limited to the current CPython org still supported releases.

Does this need to go in the CPython project and integrate with its internals such as pgen.c or pgen2?  I don't know.  From my perspective this could be a PyPI project.  Even if it seems odd that we have stdlib ast and lib2to3.pgen2 modules and pgen internal to CPython; at some point those could be seen as implementation details and made private in favor of tool application code using a canonical ponycorn thing on PyPI.  The important part is that it is maintained and kept up to date with future language grammar changes while maintaining "backwards grammar compatibility".