Under bpo-26526, I had optimised the validation code in parser module to use the actual parser DFAs, but my code considers only the token types in the input, and doesn't distinguish between different NAMEs (e.g. different keywords).

The result is this:

Python 3.7.0 (default, Aug 22 2018, 20:50:05) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import parser
>>> parser.sequence2st(parser.suite("if True:\n pass\nelse:\n pass").totuple())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
parser.ParserError: Expected node type 305, got 11.

The fix for this bug is quite simple, and in fact, it had been languishing for 2.5 years under bpo-26415

I can easily enough extract the fix into a PR of its own, but the bigger question is: parser module had been advised against using since Python 2.5; now that it has been broken for 2.5 years, nobody even noticed. (if-else must be quite a common code construct, so anybody trying to use the module would have noticed!) So, should perhaps the module be discontinued rather than fixed?

I would be curious to hear what Pablo has to say with the new parser having landed and if there's something we should be exposing from that code to replace what's in 'parser' today? (Either w/o semantic change or a new API.)

I would be curious to hear what Pablo has to say with the new parser having landed and if there's something we should be exposing from that code to replace what's in 'parser' today? (Either w/o semantic change or a new API.)

:)

One small clarification is that the parser is the same, what has changed is the parser generator. What is exposed in the parser modules today is the parse trees (in a very raw form).

One thing we can do is expose the parser component that lib2to3/pgen2 has as a substitute/complement to the parser module (which is not exposed as part of the new pgen - I know, is confusing). This is very useful and complementary to the AST (for example, black is using a forked version of this component to obtain the CST as it can do round tripping - code->CST->NEW_CST->code). This piece is in pure Python and can read the parser tables that pgen generates. It also will have the advantage of forcing us to synchronize to the current grammar (black had to fork it among other things because the one in lib2to3 was out of date). This idea and all the challenges are already been discussed here:

https://bugs.python.org/issue33337

The major problem with the parser module is that is unsynchronized with the actual parser, it has a very raw API and is moderately unmaintained (as this bug reveals). We would need to evaluate if we want to spend effort into synchronizing them, deprecating completely the parser module, substitute it with a new python version or wait until we have a completely new non-LL(1) C parser to ask these questions.

What do you think?

As a side note, the problem described in this bug was more or less foreseen. This is in the header of Modules/parsemodule.c:

• To debug parser errors like

•  "parser.ParserError: Expected node type 12, got 333."
  

• decode symbol numbers using the automatically-generated files
• Lib/symbol.h and Include/token.h.

The major problem with the parser module is that is unsynchronized with the actual parser

The parser module is "sort of" synchronised with the actual parser, in that it uses the same _PyParser_Grammar; this doesn't mean they always behave the same, as this bug shows :-)

(But before bpo-26526, it used to be much worse, with the parser module having a completely independent re-implementation of the parser.)

As a side note, the problem described in this bug was more or less foreseen. This is in the header of Modules/parsemodule.c:

Just to clarify, the bug is not about the cryptic exception message, it's about the exception firing when it shouldn't.

It's sounding like it might be worth the effort then to make lib2to3's parser not be a "hidden" thing in lib2to3, break it out as a new parser module (I have no stance on name), and then deprecate the old parser module. I think this was discussed at the last language summit when Christian proposed deprecating lib2to3 and everyone said the parser is too useful to lose.

never used the parser module nor lib2to3. Does they have any advantage over ast.parse and ast module ?

@xavier different needs; AST and CST are at different stages of compilation.

New changeset 9a0000d by Pablo Galindo in branch 'master':
bpo-36256: Fix bug in parsermodule when parsing if statements (GH-12477)
9a0000d

New changeset 00eb97b by Pablo Galindo (Miss Islington (bot)) in branch '3.7':
bpo-36256: Fix bug in parsermodule when parsing if statements (GH-12488)
00eb97b

Is it intentional that the fix is not backported to 3.6 as well?

3.6 only accepts security fixes at this point.