Note: PEP 617 is currently under review by the Steering Council, but if they approve we'd like to get it into alpha 6, and reviews are welcome (even though we're still finagling some corner cases of the implementation).

Once PR 19503 will be merged, I would be interested to see if setjmp()/longjmp() can be avoided. I'm scared by these functions: they require that all functions in between setjmp() and longjmp() call stack have no state and don't need any cleanup at exit.

VIctor, you were very right about longjmp. See https://github.com/we-like-parsers/cpython/pull/119

New changeset c5fc15685202cda73f7c3f5c6f299b0945f58508 by Pablo Galindo in branch 'master':
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
https://github.com/python/cpython/commit/c5fc15685202cda73f7c3f5c6f299b0945f58508

New changeset 458004bf7914f96b20bb76bc3584718cf83f652e by Pablo Galindo in branch 'master':
bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)
https://github.com/python/cpython/commit/458004bf7914f96b20bb76bc3584718cf83f652e

> bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)

Pablo told me in private that he plans to add a buildbot worker to test the old parser. So this change is fine.

Looks like the build changes do not work with a build directory outside of the source directory:

mkdir build && cd build && ../configure && make

gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include    -DPy_BUILD_CORE -o Parser/pegen/pegen.o ../Parser/pegen/pegen.c
error: unable to open output file 'Parser/pegen/pegen.o': 'No such file or directory'
1 error generated.
make: *** [Parser/pegen/pegen.o] Error 1

Lots of build processes depend on this, like for putting out a release.

> Looks like the build changes do not work with a build directory outside of the source directory:

Opened https://github.com/python/cpython/pull/19667. Ned, could you review?

New changeset a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6 by Pablo Galindo in branch 'master':
bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667)
https://github.com/python/cpython/commit/a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6

New changeset 1def7754b7a41fe57efafaf5eff24cfa15353444 by Victor Stinner in branch 'master':
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
https://github.com/python/cpython/commit/1def7754b7a41fe57efafaf5eff24cfa15353444

> bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)

This change removes sys.flags.use_peg which was only used in tests. If someone sees a good reason, we may add it back. In the meanwhile, test.support.use_old_parser() can be used.

I didn't like the idea of having a new sys.flags.use_peg flag appears in 3.9 but disappears in 3.10.

FYI sys.flags still supports the tuple API! Example:

>>> sys.flags[9]
0

I have no idea what is this 10th member of sys.flags :-)

test_peg_generator emits compiler warnings:

https://buildbot.python.org/all/#/builders/612/builds/283

0:08:01 load avg: 1.40 [423/423/1] test_peg_generator passed (7 min 4 sec)
/tmp/tmp0vbqe8gp/parse.c: In function ‘start_rule’:
/tmp/tmp0vbqe8gp/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(1, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp0vbqe8gp/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~
/tmp/tmp54o68k3n/parse.c: In function ‘start_rule’:
/tmp/tmp54o68k3n/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(0, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp54o68k3n/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~

> test_peg_generator emits compiler warnings

These are expected. Is this a problem?

> These are expected. Is this a problem?

People and bots running tests normally expect to not see any output from each test in the default case unless there are errors.  If these are supposed to be emitted, we should find a way to hide them (and check the results, if necessary) unless a verbose option is selected.  Personally, I wouldn't consider that a showstopper for a6 but as something to be fixed by b1.

New changeset ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d by Pablo Galindo in branch 'master':
bpo-40334: Don't downcast from Py_ssize_t to int (GH-19671)
https://github.com/python/cpython/commit/ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d

> test_peg_generator emits compiler warnings:

This is fixed by https://github.com/python/cpython/pull/19672

Those warnings are legitimate: we cannot cast function pointers that have different return types from one to the other. This only happens in the test, but could happen in the grammar if we use lookahead with NAME (&NAME).

Issue40370 documents test_peg_generator breakage on an AIX buildbot.

New changeset 1df5a9e88c8df1495a2e689d71b34bf0c7d008ea by Pablo Galindo in branch 'master':
bpo-40334: Fix build errors and warnings in test_peg_generator (GH-19672)
https://github.com/python/cpython/commit/1df5a9e88c8df1495a2e689d71b34bf0c7d008ea

New changeset 8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0 by Lysandros Nikolaou in branch 'master':
bpo-40334: Suppress all output in test_peg_generator (GH-19675)
https://github.com/python/cpython/commit/8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0

GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:
https://buildbot.python.org/all/#/builders/612/builds/286

In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5709:25,
    inlined from ‘assemble’ at Python/compile.c:6048:18:
Python/compile.c:5663:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5663 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~

> GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:

Is this related to this issue? We didn't change that line

> Is this related to this issue? We didn't change that line

I can provide you access to the buildbot if you'd like to debug the issue.

> Is this related to this issue? We didn't change that line

I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.

> I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.

Given that is a compiler warning and we didn't change those lines, I would say is not related to this PR :)

New changeset ebebb6429c224c713e1c63a0b05d4840f52c7415 by Lysandros Nikolaou in branch 'master':
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
https://github.com/python/cpython/commit/ebebb6429c224c713e1c63a0b05d4840f52c7415

AMD64 Windows10 3.x:
https://buildbot.python.org/all/#builders/129/builds/825

Warning -- files was modified by test_peg_generator
  Before: []
  After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']

> Warning -- files was modified by test_peg_generator
>   Before: []
>   After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']

Currently working on this.

$ make -s regen-all
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 14, in <module>
    from typing import Final
ImportError: cannot import name 'Final'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1

> ImportError: cannot import name 'Final'

Currently, you need Python3.8+ to do 'make regen-pegen'. We can try to modify the module to not use newer features like "Final" so you can use older Pythons.

It does not work with python3.8 either.

$ make -s regen-all PYTHON_FOR_REGEN=python3.8
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1

> oduleNotFoundError: No module named 'distutils.tests'

Oh, are you using a python distribution that strips distutils? Is this Debian or Ubuntu?

I suppose we need to account for that. I will create a PR for trying to address both. That's for pointing that out, Serhiy!

"That's for pointing that out" -> "Thanks for pointing that out"

New changeset bc28805570ae3e8835a5d502ae9ab15c52449f77 by Guido van Rossum in branch 'master':
bpo-40334: Use old compiler when compile mode is func_type (GH-19692)
https://github.com/python/cpython/commit/bc28805570ae3e8835a5d502ae9ab15c52449f77

I don't see any mention of the PEP 617 in What's New in Python 3.9:
https://docs.python.org/dev/whatsnew/3.9.html

New changeset 50f28dea32c45e1a49b3bd07c874b4fa837a5e88 by Pablo Galindo in branch 'master':
bpo-40334: Allow to run make regen-pegen without distutils (GH-19684)
https://github.com/python/cpython/commit/50f28dea32c45e1a49b3bd07c874b4fa837a5e88

test_peg_generator leaks references.

Example with s390x RHEL8 Refleaks 3.x:
https://buildbot.python.org/all/#builders/536/builds/67

test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782


The following test is enough to trigger a leak:

$ ./python -m test -R 3:3 test_peg_generator -m test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules

0:00:00 load avg: 0.94 Run tests sequentially
0:00:00 load avg: 0.94 [1/1] test_peg_generator
beginning 6 repetitions
123456
......
test_peg_generator leaked [55, 55, 55] references, sum=165
test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
test_peg_generator failed

== Tests result: FAILURE ==

1 test failed:
    test_peg_generator

Total duration: 10.8 sec
Tests result: FAILURE

Yeah that's why we have a PR out to delete or skip this test. Pablo and
Lysandros will deal with it tomorrow.

On Thu, Apr 23, 2020 at 19:33 STINNER Victor <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@python.org> added the comment:
>
> test_peg_generator leaks references.
>
> Example with s390x RHEL8 Refleaks 3.x:
> https://buildbot.python.org/all/#builders/536/builds/67
>
> test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
> test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782
>
>
> The following test is enough to trigger a leak:
>
> $ ./python -m test -R 3:3 test_peg_generator -m
> test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules
>
> 0:00:00 load avg: 0.94 Run tests sequentially
> 0:00:00 load avg: 0.94 [1/1] test_peg_generator
> beginning 6 repetitions
> 123456
> ......
> test_peg_generator leaked [55, 55, 55] references, sum=165
> test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
> test_peg_generator failed
>
> == Tests result: FAILURE ==
>
> 1 test failed:
>     test_peg_generator
>
> Total duration: 10.8 sec
> Tests result: FAILURE
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40334>
> _______________________________________
>
-- 
--Guido (mobile)

New changeset 24ffe705c30e36c82940d75fd1454256634d0b3c by Lysandros Nikolaou in branch 'master':
bpo-40334: Rewrite test_c_parser to avoid memory leaks (GH-19694)
https://github.com/python/cpython/commit/24ffe705c30e36c82940d75fd1454256634d0b3c

> test_peg_generator leaks references.

This is now fixed.

Lysandros:
> This is now fixed.

Thanks, that was quick ;-)

:waves: -- seeing a lot of failures in pyflakes' testsuite around SyntaxErrors https://github.com/PyCQA/pyflakes/pull/532

(you can reproduce with `path/to/python -m unittest discover pyflakes` without needing any dependencies installed)

New changeset 0e80b561d442769631d66f1cc8813ac30f97378e by Guido van Rossum in branch 'master':
bpo-40334: Add What's New sections for PEP 617 and PEP 585 (GH-19704)
https://github.com/python/cpython/commit/0e80b561d442769631d66f1cc8813ac30f97378e

@Anthony: Presumably some of these will be unavoidable. I see you have PYPY-specific checks in your code already.

You could also help by sending us the test programs and the different output produced by the old and the new parser. You could either create a Gist and link it here, or open bug reports here: https://github.com/we-like-parsers/cpython/issues . I opened one already for you: https://github.com/we-like-parsers/cpython/issues/121

New changeset 2b74c835a7280840a853e3a9aaeb83758b13a458 by Pablo Galindo in branch 'master':
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
https://github.com/python/cpython/commit/2b74c835a7280840a853e3a9aaeb83758b13a458

New changeset b94dbd7ac34dc0c79512656eb17f6f07e09fca7a by Pablo Galindo in branch 'master':
bpo-40334: Support PyPARSE_DONT_IMPLY_DEDENT in the new parser (GH-19736)
https://github.com/python/cpython/commit/b94dbd7ac34dc0c79512656eb17f6f07e09fca7a

The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

It should either parse the correct Grammar/Tokens file, or read the content of corresponding files Lib/token.py and Lib/tokenize.py and evaluate them with eval(). See for example Tools/scripts/generate_token.py.

> The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

Very good point, Serhiy! Thanks for catching that

So there are two parts of the parser generator where we use these modules:

- For the grammar parser and the python-based generator (that generates the grammar parser) we are good because we just need the modules to parse the grammars and generate the metaparser so there is no need for those modules to be updated. Running the grammar parser (the one generated with the meta-grammar) is also fine.

- For the C generator we need the current set of "exact_token_types" and in this case we need them to be synchronized with the Tokens file. I think we can do as pgen and take the path to that file as part of the command line interface and parse it.

I will make a PR for this soon.

I just saw this.  Interesting.  Sometimes I use ast.literal_eval to read big, deeply nested data objects.  I can probably convert to JSON if necessary but it's another thing to watch out for.  I might try to benchmark some of these.

@phr: examples welcome! The bigger and nastier, the better. This *should* all be speedy but it wouldn't hurt to check.

New changeset d55133f49fe678fbf047a647aa8bb8b520410e8d by Lysandros Nikolaou in branch 'master':
bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)
https://github.com/python/cpython/commit/d55133f49fe678fbf047a647aa8bb8b520410e8d

New changeset 3d53d8756f0403eec6a4e12f183103d651bed6c5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Don't skip test_parser:test_trigget_memory_error (GH-19744)
https://github.com/python/cpython/commit/3d53d8756f0403eec6a4e12f183103d651bed6c5

New changeset 5b9f4988c94f47fa35e84f154a7b5aa17bc04722 by Pablo Galindo in branch 'master':
bpo-40334: Refactor peg_generator to receive a Tokens file when building c code (GH-19745)
https://github.com/python/cpython/commit/5b9f4988c94f47fa35e84f154a7b5aa17bc04722

On x86 Gentoo Non-Debug with X 3.x buildbot, test_peg_generator timed out twice after 15 minutes (first time when tests are run in parallel, then when the test is run sequentially).

Maybe the buildbot worker is just slow. On a previous build (749), test_peg_generator timed out after 15 min (parallel run), but then passed in 12 minutes when re-run in verbose mode.

The buildbot worker should get a larger timeout. I suggest to increase the timeout, or the test should be optimized to take less time.

Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.

--

The failed build 750:
https://buildbot.python.org/all/#/builders/188/builds/750

0:58:45 load avg: 4.59 [423/423/2] test_peg_generator crashed (Exit code 1)
(... many compiler warnings ...)
Timeout (0:15:00)!
Thread 0xb7bba700 (most recent call first):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1873 in _try_wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1915 in _wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1185 in wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 75 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 89 in compile_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 101 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 76 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 79 in run_test
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 240 in test_return_stmt_noexpr_action
  (...)

> Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.


Done: https://github.com/python/buildmaster-config/pull/190

It seems like building Python on Windows 64-bit emits a few warnings:

D:\a\1\s\Parser\pegen\pegen.c(610,91): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(611,52): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(612,103): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]

The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):

    clear = "NORMAL"
    print(dict(state=clear, bg="#d00" if clear else"#fca"))

Well, the code looks like a typo error... but it worked previously. Not sure if we should keep the SyntaxError or not. Fixing the code is trivial: see PR 19777 attached to bpo-40431; replace >else"#fca"< with >else "#fca"<.

> The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):


This is not due to the new parser but due to https://bugs.python.org/issue40246. Please, report it there as well (you can reopen that issue for further discussion).

New changeset 37af21b667a9f41437b5b8e451497d7725016df5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix shifting of nested f-strings in the new parser (GH-19771)
https://github.com/python/cpython/commit/37af21b667a9f41437b5b8e451497d7725016df5

New changeset 2208134918ee673451e4fc525bbeab71142d794a by Pablo Galindo in branch 'master':
bpo-40334: Explicitly cast to int in pegen.c to fix a compiler warning (GH-19779)
https://github.com/python/cpython/commit/2208134918ee673451e4fc525bbeab71142d794a

New changeset 6d6508765514c7c10719478a0430f5e47c9a96ac by Lysandros Nikolaou in branch 'master':
bpo-40334: Disallow invalid single statements in the new parser (GH-19774)
https://github.com/python/cpython/commit/6d6508765514c7c10719478a0430f5e47c9a96ac

New changeset 4db245ee9ddbe6c53d375de59a35ff59dea2a8e0 by Pablo Galindo in branch 'master':
bpo-40334: refactor and cleanup for the PEG generators (GH-19775)
https://github.com/python/cpython/commit/4db245ee9ddbe6c53d375de59a35ff59dea2a8e0

New changeset 69e802ed812e38cb68a4ab74af64b4f719b6cc78 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix test_peg_parser to actually use the old parser (GH-19778)
https://github.com/python/cpython/commit/69e802ed812e38cb68a4ab74af64b4f719b6cc78

New changeset c001c09e9059ba04bc088349cd87a1374dc0754f by Guido van Rossum in branch 'master':
bpo-40334: Support type comments (GH-19780)
https://github.com/python/cpython/commit/c001c09e9059ba04bc088349cd87a1374dc0754f

Warnings found by pyflakes:

Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:115:31 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:116:25 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:42 undefined name 'stmt'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:48 undefined name 'blocks'

New changeset 3e0a6f37dfdd595be737baae00ec0e036a912615 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add support for feature_version in new PEG parser (GH-19827)
https://github.com/python/cpython/commit/3e0a6f37dfdd595be737baae00ec0e036a912615

> Warnings found by pyflakes:

Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.

New changeset b796b3fb48283412d3caf52323c69690e5818d3d by Pablo Galindo in branch 'master':
bpo-40334: Simplify type handling in the PEG c_generator (GH-19818)
https://github.com/python/cpython/commit/b796b3fb48283412d3caf52323c69690e5818d3d

While PR 19786 makes the PEG generator working on 3.6, it fails on 3.7. Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.

>  Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.

Oh sorry. I analyzed dozens of pyflakes warning. When I saw "parser", I expected the issue to be new and so introduced by this bpo. Sorry about that ;-)

New changeset 76c1b4d5c5a610c09943e1ee7ae18f1957804730 by Batuhan Taskaya in branch 'master':
bpo-40334: Improve column offsets for thrown syntax errors by Pegen (GH-19782)
https://github.com/python/cpython/commit/76c1b4d5c5a610c09943e1ee7ae18f1957804730

> Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.

Thanks, Serhiy, will submit a PR soon!

New changeset ea7297cf8f1aad4df8921a3d81a75118511afe77 by Pablo Galindo in branch 'master':
bpo-40334: unskip test_function_type in test_unparse with the new parser (GH-19837)
https://github.com/python/cpython/commit/ea7297cf8f1aad4df8921a3d81a75118511afe77

New changeset d955241469c18c946924dba79c18a9ef200391ad by Pablo Galindo in branch 'master':
bpo-40334: Correct return value of func_type_comment (GH-19833)
https://github.com/python/cpython/commit/d955241469c18c946924dba79c18a9ef200391ad

New changeset 3941d9700b2a272689cb8a8435b5c60a1466ef79 by Guido van Rossum in branch 'master':
bpo-40334: Refactor lambda_parameters similar to parameters (GH-19830)
https://github.com/python/cpython/commit/3941d9700b2a272689cb8a8435b5c60a1466ef79

New changeset 03b7642265e65f198682f22648dbe6cf4fff9835 by Lysandros Nikolaou in branch 'master':
bpo-40334: Make the PyPegen* and PyParser* APIs more consistent (GH-19839)
https://github.com/python/cpython/commit/03b7642265e65f198682f22648dbe6cf4fff9835

New changeset 7ba08ff7b41911f972d0750e068a2270e0dbd68f by Pablo Galindo in branch 'master':
bpo-40334: use the TOKENS file when checking dangling rules (GH-19849)
https://github.com/python/cpython/commit/7ba08ff7b41911f972d0750e068a2270e0dbd68f

New changeset d2baff4301387e232495491f7291903cc1217d21 by Pablo Galindo in branch 'master':
bpo-40334: regenerate metaparser as part of regen-all (GH-19854)
https://github.com/python/cpython/commit/d2baff4301387e232495491f7291903cc1217d21

Easy issue: the Python.gram file still has a comment saying something like

# Simplified grammar

We also get:

  File "/usr/lib/python3.9/site-packages/dnf/comps.py", line 111
    return'.'.join(lcl)
         ^
SyntaxError: invalid string prefix

This isn't related to this issue, but to bpo-40246. Please report it there as well.

Ah, sorry, I got it mixed up. Reported there.

New changeset 7f06af684a1882fdb19d20650825948b1d7996e5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Set error_indicator in _PyPegen_raise_error (GH-19887)
https://github.com/python/cpython/commit/7f06af684a1882fdb19d20650825948b1d7996e5

New changeset e10e7c771bf06112c4a311e0ef6b8af6423b0cca by Lysandros Nikolaou in branch 'master':
bpo-40334: Spacialized error message for invalid args after bare '*' (GH-19865)
https://github.com/python/cpython/commit/e10e7c771bf06112c4a311e0ef6b8af6423b0cca

New changeset 999ec9ab6af536cc2666a0847ec02331aaf00416 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add type to the assignment rule in the grammar file (GH-19963)
https://github.com/python/cpython/commit/999ec9ab6af536cc2666a0847ec02331aaf00416

New changeset 99db2a1db7a9b468a0ce8377d579f78fa03a2a34 by Pablo Galindo in branch 'master':
bpo-40334: Allow trailing comma in parenthesised context managers (GH-19964)
https://github.com/python/cpython/commit/99db2a1db7a9b468a0ce8377d579f78fa03a2a34

New changeset 470aac4d8e76556bd8f820f3f3928dca2b4d2849 by Pablo Galindo in branch 'master':
bpo-40334: Generate comments in the parser code to improve debugging (GH-19966)
https://github.com/python/cpython/commit/470aac4d8e76556bd8f820f3f3928dca2b4d2849

New changeset 2f37c355ab0e9ec9c1753985d27c41fa0bd719b9 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix error location upon parsing an invalid string literal (GH-19962)
https://github.com/python/cpython/commit/2f37c355ab0e9ec9c1753985d27c41fa0bd719b9

New changeset 4638c6429575bd6de26b12b2af5df74d6568b553 by Lysandros Nikolaou in branch 'master':
bpo-40334: Error message for invalid default args in function call (GH-19973)
https://github.com/python/cpython/commit/4638c6429575bd6de26b12b2af5df74d6568b553

New changeset ac7a92cc0a821699df48bc2e30a02c25d6338f78 by Pablo Galindo in branch 'master':
bpo-40334: Avoid collisions between parser variables and grammar variables (GH-19987)
https://github.com/python/cpython/commit/ac7a92cc0a821699df48bc2e30a02c25d6338f78

FWIW I propose that we add another Misc/NEWS for the same issue summarizing what happened between alpha6 and beta1. (A lot!)

There is some discrepancy with the codeop module when running with the new parser:

./python -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"

(No error)

./python -Xoldparser -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"
...
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax

Thanks Pablo for replying to my tweet, a couple of other non-failing case in 
https://bugs.python.org/issue40585 that I filled concurrently to your message.

(also using this occasion to thanks all of you for your work on the new parser)

Seem to not be in the new parser but simply in codeop in particular `def _maybe_compile`

The logic seem weird (but weird logic usually have a reason), it try to compile thrice by appending many `\n`.

1) Why do that and not return the first successful compile directly ? I'm not sure. 
2) It does compare the repr of error when  compile(source + "\n") and compile(source+'\n\n') and only raise if both are identical (which they are not in the new parser, they differ by `\n`...)

SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b')) 
vs
SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b\n'))


This logic seem to go back to the 2000s.

It definitely has its reasons -- there are pernicious edge cases. Probably revealed by the tests.

New changeset 27c0d9b54abaa4112d5a317b8aa78b39ad60a808 by Shantanu in branch 'master':
bpo-40334: produce specialized errors for invalid del targets (GH-19911)
https://github.com/python/cpython/commit/27c0d9b54abaa4112d5a317b8aa78b39ad60a808

New changeset a15c9b3a0524e5ca0434d2ad11076677824af941 by Lysandros Nikolaou in branch 'master':
bpo-40334: Always show the caret on SyntaxErrors (GH-20050)
https://github.com/python/cpython/commit/a15c9b3a0524e5ca0434d2ad11076677824af941

New changeset 16ab07063cb564c1937714bd39d6915172f005b5 by Pablo Galindo in branch 'master':
bpo-40334: Correctly identify invalid target in assignment errors (GH-20076)
https://github.com/python/cpython/commit/16ab07063cb564c1937714bd39d6915172f005b5

New changeset 2c8cd06afe8e0abb52367f85978f19b88e2df53e by Lysandros Nikolaou in branch 'master':
bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)
https://github.com/python/cpython/commit/2c8cd06afe8e0abb52367f85978f19b88e2df53e

One minor nit:  We're getting a nuisance compiler warning:

test_peg_generator passed (2 min 40 sec)
/var/folders/1b/3499qp8s7xv5w0fvjcgl45100000gn/T/tmp90rifd5z/parse.c:97:19: warning: code will never be executed [-Wunreachable-code]
        p->mark = _mark;
                  ^~~~~
1 warning generated.

I'm not getting any compiler warnings on macOS (clang) and on Ubuntu (gcc) and I can't find any related warnings on the Windows Github Actions logs either. Could you maybe post a verbose log of test_peg_generator, Raymond?

New changeset 75b863aa97016c6813709eb620c43295f84dd51f by Lysandros Nikolaou in branch 'master':
bpo-40334: Reproduce error message for type comments on bare '*' in the new parser (GH-20151)
https://github.com/python/cpython/commit/75b863aa97016c6813709eb620c43295f84dd51f

New changeset f50516e6a978ee694232512399dd1ab47aaebab1 by Batuhan Taskaya in branch 'master':
bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)
https://github.com/python/cpython/commit/f50516e6a978ee694232512399dd1ab47aaebab1

New changeset ae145833025b0156ee2a28219e3370f3b27b2a36 by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/ae145833025b0156ee2a28219e3370f3b27b2a36

New changeset 55c89235247d9dbe8a4463c9c64edc7e48826a44 by Miss Islington (bot) in branch '3.9':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/55c89235247d9dbe8a4463c9c64edc7e48826a44

New changeset cba503151056b448b7a3730dc36ef6655550ade5 by Batuhan Taskaya in branch 'master':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/cba503151056b448b7a3730dc36ef6655550ade5

New changeset 82c274e3ba7d011e93805f1552e90baea1752cf1 by Miss Islington (bot) in branch '3.9':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/82c274e3ba7d011e93805f1552e90baea1752cf1

New changeset 01ece63d42b830df106948db0aefa6c1ba24416a by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages on invalid targets (GH-20106)
https://github.com/python/cpython/commit/01ece63d42b830df106948db0aefa6c1ba24416a

New changeset a5442b26f46f1073d1eb78895d554be520105ecb by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) (GH-20973)
https://github.com/python/cpython/commit/a5442b26f46f1073d1eb78895d554be520105ecb

All parts of the implementation are now in. New issues should be opened for any potential bugs.