classification
Title: PEP 617: new PEG-based parser
Type: enhancement Stage: resolved
Components: Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, BTaskaya, gvanrossum, hauntsaninja, lys.nikolaou, mbussonn, miss-islington, ned.deily, pablogsal, rhettinger, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-04-19 20:16 by gvanrossum, last changed 2020-06-19 07:54 by lys.nikolaou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19503 merged pablogsal, 2020-04-19 20:21
PR 19664 merged pablogsal, 2020-04-22 22:34
PR 19666 merged pablogsal, 2020-04-22 22:54
PR 19667 merged pablogsal, 2020-04-22 23:42
PR 19669 merged lys.nikolaou, 2020-04-23 00:07
PR 19670 merged vstinner, 2020-04-23 00:07
PR 19668 merged pablogsal, 2020-04-23 00:12
PR 19671 merged pablogsal, 2020-04-23 01:34
PR 19672 merged pablogsal, 2020-04-23 02:23
PR 19675 merged lys.nikolaou, 2020-04-23 12:00
PR 19684 merged pablogsal, 2020-04-23 19:06
PR 19692 merged gvanrossum, 2020-04-23 22:21
PR 19694 merged lys.nikolaou, 2020-04-23 23:05
PR 19695 merged gvanrossum, 2020-04-23 23:09
PR 19704 merged gvanrossum, 2020-04-24 18:26
PR 19721 merged pablogsal, 2020-04-26 20:36
PR 19723 merged pablogsal, 2020-04-26 22:42
PR 19736 merged pablogsal, 2020-04-27 17:13
PR 19743 merged lys.nikolaou, 2020-04-27 22:45
PR 19744 merged lys.nikolaou, 2020-04-27 22:58
PR 19745 closed pablogsal, 2020-04-28 00:15
PR 19771 merged lys.nikolaou, 2020-04-28 21:34
PR 19774 merged lys.nikolaou, 2020-04-28 23:07
PR 19775 merged pablogsal, 2020-04-28 23:23
PR 19778 merged lys.nikolaou, 2020-04-28 23:36
PR 19779 merged pablogsal, 2020-04-28 23:55
PR 19780 merged gvanrossum, 2020-04-29 00:18
PR 19782 merged BTaskaya, 2020-04-29 01:34
PR 19786 closed serhiy.storchaka, 2020-04-29 07:22
PR 19818 merged pablogsal, 2020-04-30 17:23
PR 19827 merged lys.nikolaou, 2020-05-01 00:33
PR 19828 merged gvanrossum, 2020-05-01 02:42
PR 19830 merged gvanrossum, 2020-05-01 02:57
PR 19833 merged pablogsal, 2020-05-01 10:55
PR 19837 merged pablogsal, 2020-05-01 14:34
PR 19839 merged lys.nikolaou, 2020-05-01 16:32
PR 19849 merged pablogsal, 2020-05-01 21:26
PR 19854 merged pablogsal, 2020-05-02 03:56
PR 19865 merged lys.nikolaou, 2020-05-02 18:38
PR 19887 merged lys.nikolaou, 2020-05-03 20:21
PR 19911 merged hauntsaninja, 2020-05-04 21:10
PR 19962 merged lys.nikolaou, 2020-05-06 16:19
PR 19963 merged lys.nikolaou, 2020-05-06 17:42
PR 19964 merged pablogsal, 2020-05-06 17:52
PR 19966 merged pablogsal, 2020-05-06 18:47
PR 19969 open pablogsal, 2020-05-06 21:00
PR 19973 merged lys.nikolaou, 2020-05-07 00:33
PR 19987 merged pablogsal, 2020-05-08 02:45
PR 20003 merged lys.nikolaou, 2020-05-08 15:08
PR 20020 closed pablogsal, 2020-05-10 00:37
PR 20050 merged lys.nikolaou, 2020-05-12 01:29
PR 20076 merged pablogsal, 2020-05-13 22:23
PR 20106 merged lys.nikolaou, 2020-05-15 13:03
PR 20151 merged lys.nikolaou, 2020-05-17 11:50
PR 20153 merged lys.nikolaou, 2020-05-17 15:28
PR 20296 merged BTaskaya, 2020-05-21 18:51
PR 20307 merged miss-islington, 2020-05-22 00:57
PR 20367 merged BTaskaya, 2020-05-24 21:53
PR 20368 merged miss-islington, 2020-05-24 22:20
PR 20973 merged lys.nikolaou, 2020-06-18 23:31
Messages (110)
msg366803 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-19 20:16
Note: PEP 617 is currently under review by the Steering Council, but if they approve we'd like to get it into alpha 6, and reviews are welcome (even though we're still finagling some corner cases of the implementation).
msg367009 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 15:16
Once PR 19503 will be merged, I would be interested to see if setjmp()/longjmp() can be avoided. I'm scared by these functions: they require that all functions in between setjmp() and longjmp() call stack have no state and don't need any cleanup at exit.
msg367050 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-22 21:07
VIctor, you were very right about longjmp. See https://github.com/we-like-parsers/cpython/pull/119
msg367057 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 22:29
New changeset c5fc15685202cda73f7c3f5c6f299b0945f58508 by Pablo Galindo in branch 'master':
bpo-40334: PEP 617 implementation: New PEG parser for CPython (GH-19503)
https://github.com/python/cpython/commit/c5fc15685202cda73f7c3f5c6f299b0945f58508
msg367059 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 23:13
New changeset 458004bf7914f96b20bb76bc3584718cf83f652e by Pablo Galindo in branch 'master':
bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)
https://github.com/python/cpython/commit/458004bf7914f96b20bb76bc3584718cf83f652e
msg367060 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-22 23:19
> bpo-40334: Fix errors in parse_string.c with old compilers (GH-19666)

Pablo told me in private that he plans to add a buildbot worker to test the old parser. So this change is fine.
msg367061 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-22 23:21
Looks like the build changes do not work with a build directory outside of the source directory:

mkdir build && cd build && ../configure && make

gcc -c -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall    -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I../Include/internal -IObjects -IInclude -IPython -I. -I../Include    -DPy_BUILD_CORE -o Parser/pegen/pegen.o ../Parser/pegen/pegen.c
error: unable to open output file 'Parser/pegen/pegen.o': 'No such file or directory'
1 error generated.
make: *** [Parser/pegen/pegen.o] Error 1

Lots of build processes depend on this, like for putting out a release.
msg367063 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-22 23:42
> Looks like the build changes do not work with a build directory outside of the source directory:

Opened https://github.com/python/cpython/pull/19667. Ned, could you review?
msg367064 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 00:38
New changeset a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6 by Pablo Galindo in branch 'master':
bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667)
https://github.com/python/cpython/commit/a25f3c4c8f7d4878918ce1d3d67db40ae255ccc6
msg367066 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:03
New changeset 1def7754b7a41fe57efafaf5eff24cfa15353444 by Victor Stinner in branch 'master':
bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)
https://github.com/python/cpython/commit/1def7754b7a41fe57efafaf5eff24cfa15353444
msg367067 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:09
> bpo-40334: Rename PyConfig.use_peg to _use_peg_parser (GH-19670)

This change removes sys.flags.use_peg which was only used in tests. If someone sees a good reason, we may add it back. In the meanwhile, test.support.use_old_parser() can be used.

I didn't like the idea of having a new sys.flags.use_peg flag appears in 3.9 but disappears in 3.10.

FYI sys.flags still supports the tuple API! Example:

>>> sys.flags[9]
0

I have no idea what is this 10th member of sys.flags :-)
msg367068 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 01:15
test_peg_generator emits compiler warnings:

https://buildbot.python.org/all/#/builders/612/builds/283

0:08:01 load avg: 1.40 [423/423/1] test_peg_generator passed (7 min 4 sec)
/tmp/tmp0vbqe8gp/parse.c: In function ‘start_rule’:
/tmp/tmp0vbqe8gp/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(1, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp0vbqe8gp/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~
/tmp/tmp54o68k3n/parse.c: In function ‘start_rule’:
/tmp/tmp54o68k3n/parse.c:32:35: warning: passing argument 2 of ‘_PyPegen_lookahead’ from incompatible pointer type [-Wincompatible-pointer-types]
   32 |             _PyPegen_lookahead(0, _PyPegen_name_token, p)
      |                                   ^~~~~~~~~~~~~~~~~~~
      |                                   |
      |                                   struct _expr * (*)(Parser *)
In file included from /tmp/tmp54o68k3n/parse.c:2:
/home/buildbot/buildarea/3.x.cstratak-fedora-rawhide-aarch64.lto-pgo/build/Parser/pegen/pegen.h:90:36: note: expected ‘void * (*)(Parser *)’ but argument is of type ‘struct _expr * (*)(Parser *)’
   90 | int _PyPegen_lookahead(int, void *(func)(Parser *), Parser *);
      |                             ~~~~~~~^~~~~~~~~~~~~~~
msg367069 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-23 01:18
> test_peg_generator emits compiler warnings

These are expected. Is this a problem?
msg367071 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-23 01:55
> These are expected. Is this a problem?

People and bots running tests normally expect to not see any output from each test in the default case unless there are errors.  If these are supposed to be emitted, we should find a way to hide them (and check the results, if necessary) unless a verbose option is selected.  Personally, I wouldn't consider that a showstopper for a6 but as something to be fixed by b1.
msg367073 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:43
New changeset ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d by Pablo Galindo in branch 'master':
bpo-40334: Don't downcast from Py_ssize_t to int (GH-19671)
https://github.com/python/cpython/commit/ee40e4b8563e6e1bc2bfb267da5ffc9a2293318d
msg367075 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:51
> test_peg_generator emits compiler warnings:

This is fixed by https://github.com/python/cpython/pull/19672
msg367076 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 02:52
Those warnings are legitimate: we cannot cast function pointers that have different return types from one to the other. This only happens in the test, but could happen in the grammar if we use lookahead with NAME (&NAME).
msg367084 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2020-04-23 06:52
Issue40370 documents test_peg_generator breakage on an AIX buildbot.
msg367097 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 11:42
New changeset 1df5a9e88c8df1495a2e689d71b34bf0c7d008ea by Pablo Galindo in branch 'master':
bpo-40334: Fix build errors and warnings in test_peg_generator (GH-19672)
https://github.com/python/cpython/commit/1df5a9e88c8df1495a2e689d71b34bf0c7d008ea
msg367105 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 12:22
New changeset 8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0 by Lysandros Nikolaou in branch 'master':
bpo-40334: Suppress all output in test_peg_generator (GH-19675)
https://github.com/python/cpython/commit/8d1cbfffea7a5dbf7a3c60b066a2c2120ef08cd0
msg367112 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 14:05
GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:
https://buildbot.python.org/all/#/builders/612/builds/286

In function ‘assemble_lnotab’,
    inlined from ‘assemble_emit’ at Python/compile.c:5709:25,
    inlined from ‘assemble’ at Python/compile.c:6048:18:
Python/compile.c:5663:19: warning: writing 1 byte into a region of size 0 [-Wstringop-overflow=]
 5663 |         *lnotab++ = k;
      |         ~~~~~~~~~~^~~
msg367113 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 14:10
> GCC warning on aarch64 Fedora Rawhide LTO + PGO 3.x buildbot:

Is this related to this issue? We didn't change that line
msg367114 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-04-23 14:17
> Is this related to this issue? We didn't change that line

I can provide you access to the buildbot if you'd like to debug the issue.
msg367115 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 14:28
> Is this related to this issue? We didn't change that line

I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.
msg367121 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 15:17
> I don't know. It's just that I spotted this compiler warnining while reviewing recent buildbot failures.

Given that is a compiler warning and we didn't change those lines, I would say is not related to this PR :)
msg367126 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 15:36
New changeset ebebb6429c224c713e1c63a0b05d4840f52c7415 by Lysandros Nikolaou in branch 'master':
bpo-40334: Improve various PEG-Parser related stuff (GH-19669)
https://github.com/python/cpython/commit/ebebb6429c224c713e1c63a0b05d4840f52c7415
msg367135 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 18:06
AMD64 Windows10 3.x:
https://buildbot.python.org/all/#builders/129/builds/825

Warning -- files was modified by test_peg_generator
  Before: []
  After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']
msg367136 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-23 18:12
> Warning -- files was modified by test_peg_generator
>   Before: []
>   After:  ['parse_d.cp39-win_amd64.pdb', 'vc140.pdb']

Currently working on this.
msg367137 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-23 18:13
$ make -s regen-all
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 14, in <module>
    from typing import Final
ImportError: cannot import name 'Final'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1
msg367139 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:38
> ImportError: cannot import name 'Final'

Currently, you need Python3.8+ to do 'make regen-pegen'. We can try to modify the module to not use newer features like "Final" so you can use older Pythons.
msg367140 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-23 18:42
It does not work with python3.8 either.

$ make -s regen-all PYTHON_FOR_REGEN=python3.8
./Include/opcode.h.new regenerated from ./Lib/opcode.py
Jump table written into ./Python/opcode_targets.h.new
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Error in sys.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/apport_python_hook.py", line 63, in apport_excepthook
    from apport.fileutils import likely_packaged, get_recent_crashes
  File "/usr/lib/python3/dist-packages/apport/__init__.py", line 5, in <module>
    from apport.report import Report
  File "/usr/lib/python3/dist-packages/apport/report.py", line 30, in <module>
    import apport.fileutils
  File "/usr/lib/python3/dist-packages/apport/fileutils.py", line 23, in <module>
    from apport.packaging_impl import impl as packaging
  File "/usr/lib/python3/dist-packages/apport/packaging_impl.py", line 24, in <module>
    import apt
  File "/usr/lib/python3/dist-packages/apt/__init__.py", line 23, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'

Original exception was:
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 192, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/__main__.py", line 16, in <module>
    from pegen.build import build_parser_and_generator
  File "/home/serhiy/py/cpython/Tools/peg_generator/pegen/build.py", line 13, in <module>
    from distutils.tests.support import fixup_build_ext
ModuleNotFoundError: No module named 'distutils.tests'
Makefile:837: recipe for target 'regen-pegen' failed
make: *** [regen-pegen] Error 1
msg367141 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:48
> oduleNotFoundError: No module named 'distutils.tests'

Oh, are you using a python distribution that strips distutils? Is this Debian or Ubuntu?

I suppose we need to account for that. I will create a PR for trying to address both. That's for pointing that out, Serhiy!
msg367142 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 18:49
"That's for pointing that out" -> "Thanks for pointing that out"
msg367155 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-23 22:43
New changeset bc28805570ae3e8835a5d502ae9ab15c52449f77 by Guido van Rossum in branch 'master':
bpo-40334: Use old compiler when compile mode is func_type (GH-19692)
https://github.com/python/cpython/commit/bc28805570ae3e8835a5d502ae9ab15c52449f77
msg367158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 22:57
I don't see any mention of the PEP 617 in What's New in Python 3.9:
https://docs.python.org/dev/whatsnew/3.9.html
msg367160 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-23 23:53
New changeset 50f28dea32c45e1a49b3bd07c874b4fa837a5e88 by Pablo Galindo in branch 'master':
bpo-40334: Allow to run make regen-pegen without distutils (GH-19684)
https://github.com/python/cpython/commit/50f28dea32c45e1a49b3bd07c874b4fa837a5e88
msg367170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-24 02:33
test_peg_generator leaks references.

Example with s390x RHEL8 Refleaks 3.x:
https://buildbot.python.org/all/#builders/536/builds/67

test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782


The following test is enough to trigger a leak:

$ ./python -m test -R 3:3 test_peg_generator -m test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules

0:00:00 load avg: 0.94 Run tests sequentially
0:00:00 load avg: 0.94 [1/1] test_peg_generator
beginning 6 repetitions
123456
......
test_peg_generator leaked [55, 55, 55] references, sum=165
test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
test_peg_generator failed

== Tests result: FAILURE ==

1 test failed:
    test_peg_generator

Total duration: 10.8 sec
Tests result: FAILURE
msg367171 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-24 03:56
Yeah that's why we have a PR out to delete or skip this test. Pablo and
Lysandros will deal with it tomorrow.

On Thu, Apr 23, 2020 at 19:33 STINNER Victor <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@python.org> added the comment:
>
> test_peg_generator leaks references.
>
> Example with s390x RHEL8 Refleaks 3.x:
> https://buildbot.python.org/all/#builders/536/builds/67
>
> test_peg_generator leaked [24246, 24242, 24246] references, sum=72734
> test_peg_generator leaked [10928, 10926, 10928] memory blocks, sum=32782
>
>
> The following test is enough to trigger a leak:
>
> $ ./python -m test -R 3:3 test_peg_generator -m
> test.test_peg_generator.test_c_parser.TestCParser.test_error_in_rules
>
> 0:00:00 load avg: 0.94 Run tests sequentially
> 0:00:00 load avg: 0.94 [1/1] test_peg_generator
> beginning 6 repetitions
> 123456
> ......
> test_peg_generator leaked [55, 55, 55] references, sum=165
> test_peg_generator leaked [19, 19, 19] memory blocks, sum=57
> test_peg_generator failed
>
> == Tests result: FAILURE ==
>
> 1 test failed:
>     test_peg_generator
>
> Total duration: 10.8 sec
> Tests result: FAILURE
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40334>
> _______________________________________
>
-- 
--Guido (mobile)
msg367190 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-24 13:51
New changeset 24ffe705c30e36c82940d75fd1454256634d0b3c by Lysandros Nikolaou in branch 'master':
bpo-40334: Rewrite test_c_parser to avoid memory leaks (GH-19694)
https://github.com/python/cpython/commit/24ffe705c30e36c82940d75fd1454256634d0b3c
msg367191 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-04-24 13:53
> test_peg_generator leaks references.

This is now fixed.
msg367196 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-24 14:56
Lysandros:
> This is now fixed.

Thanks, that was quick ;-)
msg367237 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2020-04-24 23:09
:waves: -- seeing a lot of failures in pyflakes' testsuite around SyntaxErrors https://github.com/PyCQA/pyflakes/pull/532

(you can reproduce with `path/to/python -m unittest discover pyflakes` without needing any dependencies installed)
msg367243 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-25 00:20
New changeset 0e80b561d442769631d66f1cc8813ac30f97378e by Guido van Rossum in branch 'master':
bpo-40334: Add What's New sections for PEP 617 and PEP 585 (GH-19704)
https://github.com/python/cpython/commit/0e80b561d442769631d66f1cc8813ac30f97378e
msg367246 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-25 00:45
@Anthony: Presumably some of these will be unavoidable. I see you have PYPY-specific checks in your code already.

You could also help by sending us the test programs and the different output produced by the old and the new parser. You could either create a Gist and link it here, or open bug reports here: https://github.com/we-like-parsers/cpython/issues . I opened one already for you: https://github.com/we-like-parsers/cpython/issues/121
msg367439 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 17:02
New changeset 2b74c835a7280840a853e3a9aaeb83758b13a458 by Pablo Galindo in branch 'master':
bpo-40334: Support CO_FUTURE_BARRY_AS_BDFL in the new parser (GH-19721)
https://github.com/python/cpython/commit/2b74c835a7280840a853e3a9aaeb83758b13a458
msg367440 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 17:36
New changeset b94dbd7ac34dc0c79512656eb17f6f07e09fca7a by Pablo Galindo in branch 'master':
bpo-40334: Support PyPARSE_DONT_IMPLY_DEDENT in the new parser (GH-19736)
https://github.com/python/cpython/commit/b94dbd7ac34dc0c79512656eb17f6f07e09fca7a
msg367448 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-27 19:21
The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

It should either parse the correct Grammar/Tokens file, or read the content of corresponding files Lib/token.py and Lib/tokenize.py and evaluate them with eval(). See for example Tools/scripts/generate_token.py.
msg367467 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-27 22:15
> The parser generator imports modules token and tokenize. It is not correct, because they are relevant to the Python version used to run the parser generator, and not to the Python version for which the parser is generated. It works currently only because there is no differences between 3.8 and 3.9, but it will fail when you add a new token or change/remove an old one.

Very good point, Serhiy! Thanks for catching that

So there are two parts of the parser generator where we use these modules:

- For the grammar parser and the python-based generator (that generates the grammar parser) we are good because we just need the modules to parse the grammars and generate the metaparser so there is no need for those modules to be updated. Running the grammar parser (the one generated with the meta-grammar) is also fine.

- For the C generator we need the current set of "exact_token_types" and in this case we need them to be synchronized with the Tokens file. I think we can do as pgen and take the path to that file as part of the command line interface and parse it.

I will make a PR for this soon.
msg367470 - (view) Author: paul rubin (phr) Date: 2020-04-27 23:52
I just saw this.  Interesting.  Sometimes I use ast.literal_eval to read big, deeply nested data objects.  I can probably convert to JSON if necessary but it's another thing to watch out for.  I might try to benchmark some of these.
msg367471 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-27 23:55
@phr: examples welcome! The bigger and nastier, the better. This *should* all be speedy but it wouldn't hurt to check.
msg367473 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 00:23
New changeset d55133f49fe678fbf047a647aa8bb8b520410e8d by Lysandros Nikolaou in branch 'master':
bpo-40334: Catch E_EOF error, when the tokenizer returns ERRORTOKEN (GH-19743)
https://github.com/python/cpython/commit/d55133f49fe678fbf047a647aa8bb8b520410e8d
msg367474 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 00:24
New changeset 3d53d8756f0403eec6a4e12f183103d651bed6c5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Don't skip test_parser:test_trigget_memory_error (GH-19744)
https://github.com/python/cpython/commit/3d53d8756f0403eec6a4e12f183103d651bed6c5
msg367521 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 12:12
New changeset 5b9f4988c94f47fa35e84f154a7b5aa17bc04722 by Pablo Galindo in branch 'master':
bpo-40334: Refactor peg_generator to receive a Tokens file when building c code (GH-19745)
https://github.com/python/cpython/commit/5b9f4988c94f47fa35e84f154a7b5aa17bc04722
msg367564 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 21:27
On x86 Gentoo Non-Debug with X 3.x buildbot, test_peg_generator timed out twice after 15 minutes (first time when tests are run in parallel, then when the test is run sequentially).

Maybe the buildbot worker is just slow. On a previous build (749), test_peg_generator timed out after 15 min (parallel run), but then passed in 12 minutes when re-run in verbose mode.

The buildbot worker should get a larger timeout. I suggest to increase the timeout, or the test should be optimized to take less time.

Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.

--

The failed build 750:
https://buildbot.python.org/all/#/builders/188/builds/750

0:58:45 load avg: 4.59 [423/423/2] test_peg_generator crashed (Exit code 1)
(... many compiler warnings ...)
Timeout (0:15:00)!
Thread 0xb7bba700 (most recent call first):
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1873 in _try_wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1915 in _wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1185 in wait
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 75 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 89 in compile_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 101 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 76 in build_extension
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 79 in run_test
  File "/buildbot/buildarea/cpython/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 240 in test_return_stmt_noexpr_action
  (...)
msg367576 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-28 23:43
> Pablo: Would you mind to increase ware-gentoo-x86 timeout? Ex: 30 min instead of 15 min.


Done: https://github.com/python/buildmaster-config/pull/190
msg367577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 23:45
It seems like building Python on Windows 64-bit emits a few warnings:

D:\a\1\s\Parser\pegen\pegen.c(610,91): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(611,52): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
D:\a\1\s\Parser\pegen\pegen.c(612,103): warning C4267: '=': conversion from 'size_t' to 'int', possible loss of data [D:\a\1\s\PCbuild\pythoncore.vcxproj]
msg367580 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-28 23:58
The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):

    clear = "NORMAL"
    print(dict(state=clear, bg="#d00" if clear else"#fca"))

Well, the code looks like a typo error... but it worked previously. Not sure if we should keep the SyntaxError or not. Fixing the code is trivial: see PR 19777 attached to bpo-40431; replace >else"#fca"< with >else "#fca"<.
msg367582 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 00:00
> The following code works on Python 3.8, but fails with SyntaxError on Python 3.9.0a6 with the old and the new parser (see bpo-40431):


This is not due to the new parser but due to https://bugs.python.org/issue40246. Please, report it there as well (you can reopen that issue for further discussion).
msg367598 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 00:43
New changeset 37af21b667a9f41437b5b8e451497d7725016df5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix shifting of nested f-strings in the new parser (GH-19771)
https://github.com/python/cpython/commit/37af21b667a9f41437b5b8e451497d7725016df5
msg367602 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 01:04
New changeset 2208134918ee673451e4fc525bbeab71142d794a by Pablo Galindo in branch 'master':
bpo-40334: Explicitly cast to int in pegen.c to fix a compiler warning (GH-19779)
https://github.com/python/cpython/commit/2208134918ee673451e4fc525bbeab71142d794a
msg367610 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 01:42
New changeset 6d6508765514c7c10719478a0430f5e47c9a96ac by Lysandros Nikolaou in branch 'master':
bpo-40334: Disallow invalid single statements in the new parser (GH-19774)
https://github.com/python/cpython/commit/6d6508765514c7c10719478a0430f5e47c9a96ac
msg367637 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 09:42
New changeset 4db245ee9ddbe6c53d375de59a35ff59dea2a8e0 by Pablo Galindo in branch 'master':
bpo-40334: refactor and cleanup for the PEG generators (GH-19775)
https://github.com/python/cpython/commit/4db245ee9ddbe6c53d375de59a35ff59dea2a8e0
msg367713 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-04-29 22:53
New changeset 69e802ed812e38cb68a4ab74af64b4f719b6cc78 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix test_peg_parser to actually use the old parser (GH-19778)
https://github.com/python/cpython/commit/69e802ed812e38cb68a4ab74af64b4f719b6cc78
msg367772 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-04-30 19:12
New changeset c001c09e9059ba04bc088349cd87a1374dc0754f by Guido van Rossum in branch 'master':
bpo-40334: Support type comments (GH-19780)
https://github.com/python/cpython/commit/c001c09e9059ba04bc088349cd87a1374dc0754f
msg367815 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-01 00:00
Warnings found by pyflakes:

Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:115:31 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:116:25 undefined name 'lines'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:42 undefined name 'stmt'
Lib/test/test_tools/test_c_analyzer/test_parser/test_declarations.py:527:48 undefined name 'blocks'
msg367822 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 03:27
New changeset 3e0a6f37dfdd595be737baae00ec0e036a912615 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add support for feature_version in new PEG parser (GH-19827)
https://github.com/python/cpython/commit/3e0a6f37dfdd595be737baae00ec0e036a912615
msg367835 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 10:12
> Warnings found by pyflakes:

Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.
msg367836 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 11:32
New changeset b796b3fb48283412d3caf52323c69690e5818d3d by Pablo Galindo in branch 'master':
bpo-40334: Simplify type handling in the PEG c_generator (GH-19818)
https://github.com/python/cpython/commit/b796b3fb48283412d3caf52323c69690e5818d3d
msg367839 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-01 12:43
While PR 19786 makes the PEG generator working on 3.6, it fails on 3.7. Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.
msg367840 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-01 12:46
>  Hummmm, "Lib/test/test_tools/test_c_analyzer/test_parser" has nothing to do with this PR. This looks like Eric Snow's c analyzer for avoiding globals and other stuff for sub interpreters.

Oh sorry. I analyzed dozens of pyflakes warning. When I saw "parser", I expected the issue to be new and so introduced by this bpo. Sorry about that ;-)
msg367843 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 13:13
New changeset 76c1b4d5c5a610c09943e1ee7ae18f1957804730 by Batuhan Taskaya in branch 'master':
bpo-40334: Improve column offsets for thrown syntax errors by Pegen (GH-19782)
https://github.com/python/cpython/commit/76c1b4d5c5a610c09943e1ee7ae18f1957804730
msg367845 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 13:45
> Seems there are more dependencies on wrong tokenizer. Please don't forget to fix this.

Thanks, Serhiy, will submit a PR soon!
msg367850 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 15:02
New changeset ea7297cf8f1aad4df8921a3d81a75118511afe77 by Pablo Galindo in branch 'master':
bpo-40334: unskip test_function_type in test_unparse with the new parser (GH-19837)
https://github.com/python/cpython/commit/ea7297cf8f1aad4df8921a3d81a75118511afe77
msg367854 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-01 15:32
New changeset d955241469c18c946924dba79c18a9ef200391ad by Pablo Galindo in branch 'master':
bpo-40334: Correct return value of func_type_comment (GH-19833)
https://github.com/python/cpython/commit/d955241469c18c946924dba79c18a9ef200391ad
msg367862 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 16:42
New changeset 3941d9700b2a272689cb8a8435b5c60a1466ef79 by Guido van Rossum in branch 'master':
bpo-40334: Refactor lambda_parameters similar to parameters (GH-19830)
https://github.com/python/cpython/commit/3941d9700b2a272689cb8a8435b5c60a1466ef79
msg367866 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 17:30
New changeset 03b7642265e65f198682f22648dbe6cf4fff9835 by Lysandros Nikolaou in branch 'master':
bpo-40334: Make the PyPegen* and PyParser* APIs more consistent (GH-19839)
https://github.com/python/cpython/commit/03b7642265e65f198682f22648dbe6cf4fff9835
msg367891 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-01 22:14
New changeset 7ba08ff7b41911f972d0750e068a2270e0dbd68f by Pablo Galindo in branch 'master':
bpo-40334: use the TOKENS file when checking dangling rules (GH-19849)
https://github.com/python/cpython/commit/7ba08ff7b41911f972d0750e068a2270e0dbd68f
msg367914 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-02 04:23
New changeset d2baff4301387e232495491f7291903cc1217d21 by Pablo Galindo in branch 'master':
bpo-40334: regenerate metaparser as part of regen-all (GH-19854)
https://github.com/python/cpython/commit/d2baff4301387e232495491f7291903cc1217d21
msg367916 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-02 05:06
Easy issue: the Python.gram file still has a comment saying something like

# Simplified grammar
msg367974 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-03 15:29
We also get:

  File "/usr/lib/python3.9/site-packages/dnf/comps.py", line 111
    return'.'.join(lcl)
         ^
SyntaxError: invalid string prefix
msg367976 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-03 15:37
This isn't related to this issue, but to bpo-40246. Please report it there as well.
msg367978 - (view) Author: Miro Hrončok (hroncok) * Date: 2020-05-03 15:48
Ah, sorry, I got it mixed up. Reported there.
msg368004 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-04 00:20
New changeset 7f06af684a1882fdb19d20650825948b1d7996e5 by Lysandros Nikolaou in branch 'master':
bpo-40334: Set error_indicator in _PyPegen_raise_error (GH-19887)
https://github.com/python/cpython/commit/7f06af684a1882fdb19d20650825948b1d7996e5
msg368034 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-04 10:58
New changeset e10e7c771bf06112c4a311e0ef6b8af6423b0cca by Lysandros Nikolaou in branch 'master':
bpo-40334: Spacialized error message for invalid args after bare '*' (GH-19865)
https://github.com/python/cpython/commit/e10e7c771bf06112c4a311e0ef6b8af6423b0cca
msg368288 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 18:11
New changeset 999ec9ab6af536cc2666a0847ec02331aaf00416 by Lysandros Nikolaou in branch 'master':
bpo-40334: Add type to the assignment rule in the grammar file (GH-19963)
https://github.com/python/cpython/commit/999ec9ab6af536cc2666a0847ec02331aaf00416
msg368302 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 21:54
New changeset 99db2a1db7a9b468a0ce8377d579f78fa03a2a34 by Pablo Galindo in branch 'master':
bpo-40334: Allow trailing comma in parenthesised context managers (GH-19964)
https://github.com/python/cpython/commit/99db2a1db7a9b468a0ce8377d579f78fa03a2a34
msg368304 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-06 22:14
New changeset 470aac4d8e76556bd8f820f3f3928dca2b4d2849 by Pablo Galindo in branch 'master':
bpo-40334: Generate comments in the parser code to improve debugging (GH-19966)
https://github.com/python/cpython/commit/470aac4d8e76556bd8f820f3f3928dca2b4d2849
msg368329 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-07 10:37
New changeset 2f37c355ab0e9ec9c1753985d27c41fa0bd719b9 by Lysandros Nikolaou in branch 'master':
bpo-40334: Fix error location upon parsing an invalid string literal (GH-19962)
https://github.com/python/cpython/commit/2f37c355ab0e9ec9c1753985d27c41fa0bd719b9
msg368330 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-07 10:44
New changeset 4638c6429575bd6de26b12b2af5df74d6568b553 by Lysandros Nikolaou in branch 'master':
bpo-40334: Error message for invalid default args in function call (GH-19973)
https://github.com/python/cpython/commit/4638c6429575bd6de26b12b2af5df74d6568b553
msg368566 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 04:34
New changeset ac7a92cc0a821699df48bc2e30a02c25d6338f78 by Pablo Galindo in branch 'master':
bpo-40334: Avoid collisions between parser variables and grammar variables (GH-19987)
https://github.com/python/cpython/commit/ac7a92cc0a821699df48bc2e30a02c25d6338f78
msg368567 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 04:40
FWIW I propose that we add another Misc/NEWS for the same issue summarizing what happened between alpha6 and beta1. (A lot!)
msg368593 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-10 22:50
There is some discrepancy with the codeop module when running with the new parser:

./python -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"

(No error)

./python -Xoldparser -c "import codeop; codeop.CommandCompiler()('raise = 2\n\n', symbol='exec')"
...
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
msg368595 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-05-10 23:07
Thanks Pablo for replying to my tweet, a couple of other non-failing case in 
https://bugs.python.org/issue40585 that I filled concurrently to your message.

(also using this occasion to thanks all of you for your work on the new parser)
msg368598 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-05-10 23:44
Seem to not be in the new parser but simply in codeop in particular `def _maybe_compile`

The logic seem weird (but weird logic usually have a reason), it try to compile thrice by appending many `\n`.

1) Why do that and not return the first successful compile directly ? I'm not sure. 
2) It does compare the repr of error when  compile(source + "\n") and compile(source+'\n\n') and only raise if both are identical (which they are not in the new parser, they differ by `\n`...)

SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b')) 
vs
SyntaxError('invalid syntax', ('<input>', 1, 6, 'def a-b\n'))


This logic seem to go back to the 2000s.
msg368599 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-11 00:10
It definitely has its reasons -- there are pernicious edge cases. Probably revealed by the tests.
msg368663 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-11 21:54
New changeset 27c0d9b54abaa4112d5a317b8aa78b39ad60a808 by Shantanu in branch 'master':
bpo-40334: produce specialized errors for invalid del targets (GH-19911)
https://github.com/python/cpython/commit/27c0d9b54abaa4112d5a317b8aa78b39ad60a808
msg368796 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-13 19:36
New changeset a15c9b3a0524e5ca0434d2ad11076677824af941 by Lysandros Nikolaou in branch 'master':
bpo-40334: Always show the caret on SyntaxErrors (GH-20050)
https://github.com/python/cpython/commit/a15c9b3a0524e5ca0434d2ad11076677824af941
msg368892 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-15 01:04
New changeset 16ab07063cb564c1937714bd39d6915172f005b5 by Pablo Galindo in branch 'master':
bpo-40334: Correctly identify invalid target in assignment errors (GH-20076)
https://github.com/python/cpython/commit/16ab07063cb564c1937714bd39d6915172f005b5
msg369090 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-17 03:19
New changeset 2c8cd06afe8e0abb52367f85978f19b88e2df53e by Lysandros Nikolaou in branch 'master':
bpo-40334: Improvements to error-handling code in the PEG parser (GH-20003)
https://github.com/python/cpython/commit/2c8cd06afe8e0abb52367f85978f19b88e2df53e
msg369166 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-05-18 03:36
One minor nit:  We're getting a nuisance compiler warning:

test_peg_generator passed (2 min 40 sec)
/var/folders/1b/3499qp8s7xv5w0fvjcgl45100000gn/T/tmp90rifd5z/parse.c:97:19: warning: code will never be executed [-Wunreachable-code]
        p->mark = _mark;
                  ^~~~~
1 warning generated.
msg369203 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-18 10:32
I'm not getting any compiler warnings on macOS (clang) and on Ubuntu (gcc) and I can't find any related warnings on the Windows Github Actions logs either. Could you maybe post a verbose log of test_peg_generator, Raymond?
msg369288 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-18 19:14
New changeset 75b863aa97016c6813709eb620c43295f84dd51f by Lysandros Nikolaou in branch 'master':
bpo-40334: Reproduce error message for type comments on bare '*' in the new parser (GH-20151)
https://github.com/python/cpython/commit/75b863aa97016c6813709eb620c43295f84dd51f
msg369536 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-21 19:57
New changeset f50516e6a978ee694232512399dd1ab47aaebab1 by Batuhan Taskaya in branch 'master':
bpo-40334: Correctly generate C parser when assigned var is None (GH-20296)
https://github.com/python/cpython/commit/f50516e6a978ee694232512399dd1ab47aaebab1
msg369553 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-22 00:56
New changeset ae145833025b0156ee2a28219e3370f3b27b2a36 by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/ae145833025b0156ee2a28219e3370f3b27b2a36
msg369554 - (view) Author: miss-islington (miss-islington) Date: 2020-05-22 01:15
New changeset 55c89235247d9dbe8a4463c9c64edc7e48826a44 by Miss Islington (bot) in branch '3.9':
bpo-40334: Produce better error messages for non-parenthesized genexps (GH-20153)
https://github.com/python/cpython/commit/55c89235247d9dbe8a4463c9c64edc7e48826a44
msg369837 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-24 22:20
New changeset cba503151056b448b7a3730dc36ef6655550ade5 by Batuhan Taskaya in branch 'master':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/cba503151056b448b7a3730dc36ef6655550ade5
msg369840 - (view) Author: miss-islington (miss-islington) Date: 2020-05-24 22:38
New changeset 82c274e3ba7d011e93805f1552e90baea1752cf1 by Miss Islington (bot) in branch '3.9':
bpo-40334: Support suppressing of multiple optional variables in Pegen (GH-20367)
https://github.com/python/cpython/commit/82c274e3ba7d011e93805f1552e90baea1752cf1
msg371841 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-18 23:10
New changeset 01ece63d42b830df106948db0aefa6c1ba24416a by Lysandros Nikolaou in branch 'master':
bpo-40334: Produce better error messages on invalid targets (GH-20106)
https://github.com/python/cpython/commit/01ece63d42b830df106948db0aefa6c1ba24416a
msg371842 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-06-19 00:04
New changeset a5442b26f46f1073d1eb78895d554be520105ecb by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40334: Produce better error messages on invalid targets (GH-20106) (GH-20973)
https://github.com/python/cpython/commit/a5442b26f46f1073d1eb78895d554be520105ecb
msg371848 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-06-19 07:54
All parts of the implementation are now in. New issues should be opened for any potential bugs.
History
Date User Action Args
2020-06-19 07:54:58lys.nikolaousetstatus: open -> closed
resolution: fixed
messages: + msg371848

stage: patch review -> resolved
2020-06-19 00:04:05pablogsalsetmessages: + msg371842
2020-06-18 23:31:17lys.nikolaousetpull_requests: + pull_request20150
2020-06-18 23:10:50pablogsalsetmessages: + msg371841
2020-05-24 22:38:51miss-islingtonsetmessages: + msg369840
2020-05-24 22:20:31miss-islingtonsetpull_requests: + pull_request19632
2020-05-24 22:20:21pablogsalsetmessages: + msg369837
2020-05-24 21:53:59BTaskayasetpull_requests: + pull_request19631
2020-05-22 01:15:02miss-islingtonsetmessages: + msg369554
2020-05-22 00:57:05miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request19578
2020-05-22 00:56:59pablogsalsetmessages: + msg369553
2020-05-21 19:57:56pablogsalsetmessages: + msg369536
2020-05-21 18:51:19BTaskayasetpull_requests: + pull_request19571
2020-05-18 19:14:53pablogsalsetmessages: + msg369288
2020-05-18 10:32:00lys.nikolaousetmessages: + msg369203
2020-05-18 03:36:21rhettingersetnosy: + rhettinger
messages: + msg369166
2020-05-17 15:28:06lys.nikolaousetpull_requests: + pull_request19456
2020-05-17 11:50:38lys.nikolaousetpull_requests: + pull_request19454
2020-05-17 03:19:27pablogsalsetmessages: + msg369090
2020-05-15 13:03:38lys.nikolaousetpull_requests: + pull_request19413
2020-05-15 01:04:57pablogsalsetmessages: + msg368892
2020-05-13 22:23:32pablogsalsetpull_requests: + pull_request19382
2020-05-13 19:36:39pablogsalsetmessages: + msg368796
2020-05-12 01:29:33lys.nikolaousetpull_requests: + pull_request19359
2020-05-11 22:11:40pablogsalsetpull_requests: - pull_request19349
2020-05-11 21:54:16gvanrossumsetmessages: + msg368663
2020-05-11 19:32:41pablogsalsetpull_requests: + pull_request19349
2020-05-11 00:10:29gvanrossumsetmessages: + msg368599
2020-05-10 23:44:03mbussonnsetmessages: + msg368598
2020-05-10 23:07:47mbussonnsetnosy: + mbussonn
messages: + msg368595
2020-05-10 22:50:12pablogsalsetmessages: + msg368593
2020-05-10 04:40:45gvanrossumsetmessages: + msg368567
2020-05-10 04:34:57gvanrossumsetmessages: + msg368566
2020-05-10 00:37:19pablogsalsetpull_requests: + pull_request19331
2020-05-08 15:08:54lys.nikolaousetpull_requests: + pull_request19315
2020-05-08 02:45:05pablogsalsetpull_requests: + pull_request19303
2020-05-07 10:44:10pablogsalsetmessages: + msg368330
2020-05-07 10:37:58pablogsalsetmessages: + msg368329
2020-05-07 00:33:02lys.nikolaousetpull_requests: + pull_request19290
2020-05-06 22:14:49pablogsalsetmessages: + msg368304
2020-05-06 21:54:42pablogsalsetmessages: + msg368302
2020-05-06 21:00:17pablogsalsetpull_requests: + pull_request19285
2020-05-06 18:47:00pablogsalsetpull_requests: + pull_request19281
2020-05-06 18:11:14pablogsalsetmessages: + msg368288
2020-05-06 17:52:03pablogsalsetpull_requests: + pull_request19280
2020-05-06 17:42:48lys.nikolaousetpull_requests: + pull_request19279
2020-05-06 16:19:26lys.nikolaousetpull_requests: + pull_request19278
2020-05-04 21:10:35hauntsaninjasetnosy: + hauntsaninja
pull_requests: + pull_request19226
2020-05-04 16:28:53vstinnersetnosy: - vstinner
2020-05-04 11:19:45hroncoksetnosy: - hroncok
2020-05-04 11:11:45cstrataksetnosy: - cstratak
2020-05-04 10:58:42pablogsalsetmessages: + msg368034
2020-05-04 00:20:12pablogsalsetmessages: + msg368004
2020-05-03 20:21:23lys.nikolaousetpull_requests: + pull_request19197
2020-05-03 15:48:15hroncoksetnosy: gvanrossum, vstinner, ned.deily, serhiy.storchaka, Anthony Sottile, cstratak, hroncok, lys.nikolaou, pablogsal, BTaskaya
messages: + msg367978
2020-05-03 15:37:11lys.nikolaousetmessages: + msg367976
2020-05-03 15:29:33hroncoksetnosy: + hroncok
messages: + msg367974
2020-05-02 20:11:56lys.nikolaousettype: resource usage -> enhancement
2020-05-02 18:38:12lys.nikolaousetpull_requests: + pull_request19179
2020-05-02 05:06:47gvanrossumsetmessages: + msg367916
2020-05-02 04:23:47pablogsalsetmessages: + msg367914
2020-05-02 03:56:27pablogsalsetpull_requests: + pull_request19170
2020-05-01 22:14:15pablogsalsetmessages: + msg367891
2020-05-01 21:26:45pablogsalsetpull_requests: + pull_request19166
2020-05-01 17:30:59pablogsalsetmessages: + msg367866
2020-05-01 16:42:10pablogsalsetmessages: + msg367862
2020-05-01 16:32:58lys.nikolaousetpull_requests: + pull_request19157
2020-05-01 15:32:16gvanrossumsetmessages: + msg367854
2020-05-01 15:02:13gvanrossumsetmessages: + msg367850
2020-05-01 14:34:28pablogsalsetpull_requests: + pull_request19155
2020-05-01 13:45:59pablogsalsetmessages: + msg367845
2020-05-01 13:13:55pablogsalsetmessages: + msg367843
2020-05-01 12:46:51vstinnersetmessages: + msg367840
2020-05-01 12:43:17serhiy.storchakasetmessages: + msg367839
2020-05-01 11:32:30pablogsalsetmessages: + msg367836
2020-05-01 10:55:18pablogsalsetpull_requests: + pull_request19152
2020-05-01 10:12:02pablogsalsetmessages: + msg367835
2020-05-01 03:27:59gvanrossumsetmessages: + msg367822
2020-05-01 02:57:51gvanrossumsetpull_requests: + pull_request19150
2020-05-01 02:42:47gvanrossumsetpull_requests: + pull_request19148
2020-05-01 00:33:07lys.nikolaousetpull_requests: + pull_request19147
2020-05-01 00:00:42vstinnersetnosy: + vstinner
messages: + msg367815
2020-04-30 19:12:32gvanrossumsetmessages: + msg367772
2020-04-30 17:23:04pablogsalsetpull_requests: + pull_request19138
2020-04-29 22:53:36pablogsalsetmessages: + msg367713
2020-04-29 09:42:31pablogsalsetmessages: + msg367637
2020-04-29 07:22:35serhiy.storchakasetpull_requests: + pull_request19109
2020-04-29 01:42:30pablogsalsetmessages: + msg367610
2020-04-29 01:35:35vstinnersetnosy: - vstinner
2020-04-29 01:34:47BTaskayasetpull_requests: + pull_request19104
2020-04-29 01:04:09pablogsalsetmessages: + msg367602
2020-04-29 00:43:54pablogsalsetmessages: + msg367598
2020-04-29 00:18:23gvanrossumsetpull_requests: + pull_request19100
2020-04-29 00:00:42pablogsalsetmessages: + msg367582
2020-04-28 23:58:07vstinnersetmessages: + msg367580
2020-04-28 23:55:34pablogsalsetpull_requests: + pull_request19099
2020-04-28 23:45:33vstinnersetmessages: + msg367577
2020-04-28 23:43:20pablogsalsetmessages: + msg367576
2020-04-28 23:36:53lys.nikolaousetpull_requests: + pull_request19098
2020-04-28 23:23:58pablogsalsetpull_requests: + pull_request19095
2020-04-28 23:07:01lys.nikolaousetpull_requests: + pull_request19094
2020-04-28 21:34:08lys.nikolaousetpull_requests: + pull_request19091
2020-04-28 21:27:47vstinnersetnosy: + vstinner
messages: + msg367564
2020-04-28 12:47:03vstinnersetnosy: - vstinner
2020-04-28 12:12:05pablogsalsetmessages: + msg367521
2020-04-28 00:26:58phrsetnosy: - phr
2020-04-28 00:24:54pablogsalsetmessages: + msg367474
2020-04-28 00:23:39pablogsalsetmessages: + msg367473
2020-04-28 00:15:11pablogsalsetpull_requests: + pull_request19067
2020-04-27 23:55:52gvanrossumsetmessages: + msg367471
2020-04-27 23:52:13phrsetnosy: + phr
messages: + msg367470
2020-04-27 22:58:04lys.nikolaousetpull_requests: + pull_request19066
2020-04-27 22:45:47lys.nikolaousetpull_requests: + pull_request19065
2020-04-27 22:15:32pablogsalsetmessages: + msg367467
2020-04-27 19:21:03serhiy.storchakasetmessages: + msg367448
2020-04-27 17:36:04pablogsalsetmessages: + msg367440
2020-04-27 17:13:52pablogsalsetpull_requests: + pull_request19058
2020-04-27 17:02:14pablogsalsetmessages: + msg367439
2020-04-26 22:42:26pablogsalsetpull_requests: + pull_request19045
2020-04-26 20:36:06pablogsalsetpull_requests: + pull_request19043
2020-04-25 00:45:55gvanrossumsetmessages: + msg367246
2020-04-25 00:20:03gvanrossumsetmessages: + msg367243
2020-04-24 23:09:21Anthony Sottilesetnosy: + Anthony Sottile
messages: + msg367237
2020-04-24 18:26:50gvanrossumsetpull_requests: + pull_request19022
2020-04-24 14:56:57vstinnersetmessages: + msg367196
2020-04-24 13:53:46lys.nikolaousetmessages: + msg367191
2020-04-24 13:51:15pablogsalsetmessages: + msg367190
2020-04-24 03:56:20gvanrossumsetmessages: + msg367171
2020-04-24 02:33:26vstinnersetmessages: + msg367170
2020-04-23 23:53:36pablogsalsetmessages: + msg367160
2020-04-23 23:09:36gvanrossumsetpull_requests: + pull_request19015
2020-04-23 23:05:04lys.nikolaousetpull_requests: + pull_request19014
2020-04-23 22:57:52vstinnersetmessages: + msg367158
2020-04-23 22:43:02gvanrossumsetmessages: + msg367155
2020-04-23 22:21:14gvanrossumsetpull_requests: + pull_request19012
2020-04-23 19:06:44pablogsalsetpull_requests: + pull_request19004
2020-04-23 18:49:34pablogsalsetmessages: + msg367142
2020-04-23 18:48:31pablogsalsetmessages: + msg367141
2020-04-23 18:42:14serhiy.storchakasetmessages: + msg367140
2020-04-23 18:38:08pablogsalsetmessages: + msg367139
2020-04-23 18:13:34serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg367137
2020-04-23 18:12:04lys.nikolaousetmessages: + msg367136
2020-04-23 18:06:49vstinnersetmessages: + msg367135
2020-04-23 15:36:14pablogsalsetmessages: + msg367126
2020-04-23 15:17:09pablogsalsetmessages: + msg367121
2020-04-23 14:28:55vstinnersetmessages: + msg367115
2020-04-23 14:17:18cstrataksetnosy: + cstratak
messages: + msg367114
2020-04-23 14:10:50pablogsalsetmessages: + msg367113
2020-04-23 14:05:48vstinnersetmessages: + msg367112
2020-04-23 12:22:35pablogsalsetmessages: + msg367105
2020-04-23 12:00:37lys.nikolaousetpull_requests: + pull_request18999
2020-04-23 11:42:17pablogsalsetmessages: + msg367097
2020-04-23 06:52:42ned.deilysetmessages: + msg367084
2020-04-23 02:52:25pablogsalsetmessages: + msg367076
2020-04-23 02:51:15pablogsalsetmessages: + msg367075
2020-04-23 02:43:11pablogsalsetmessages: + msg367073
2020-04-23 02:23:22pablogsalsetpull_requests: + pull_request18996
2020-04-23 01:55:43ned.deilysetmessages: + msg367071
2020-04-23 01:34:56pablogsalsetpull_requests: + pull_request18994
2020-04-23 01:33:17Wade Sanchezsettype: resource usage
2020-04-23 01:18:51lys.nikolaousetmessages: + msg367069
2020-04-23 01:15:55vstinnersetmessages: + msg367068
2020-04-23 01:09:20vstinnersetmessages: + msg367067
2020-04-23 01:03:27vstinnersetmessages: + msg367066
2020-04-23 00:38:15pablogsalsetmessages: + msg367064
2020-04-23 00:12:19pablogsalsetpull_requests: + pull_request18993
2020-04-23 00:07:49vstinnersetpull_requests: + pull_request18992
2020-04-23 00:07:29lys.nikolaousetpull_requests: + pull_request18991
2020-04-22 23:42:52pablogsalsetmessages: + msg367063
2020-04-22 23:42:28pablogsalsetpull_requests: + pull_request18990
2020-04-22 23:21:25ned.deilysetnosy: + ned.deily
messages: + msg367061
2020-04-22 23:19:28vstinnersetmessages: + msg367060
2020-04-22 23:13:52pablogsalsetmessages: + msg367059
2020-04-22 22:54:01pablogsalsetpull_requests: + pull_request18989
2020-04-22 22:34:23pablogsalsetpull_requests: + pull_request18988
2020-04-22 22:29:49pablogsalsetmessages: + msg367057
2020-04-22 21:07:54gvanrossumsetmessages: + msg367050
2020-04-22 15:16:53vstinnersetnosy: + vstinner
messages: + msg367009
2020-04-19 22:11:59lys.nikolaousetnosy: + lys.nikolaou
2020-04-19 20:21:23pablogsalsetkeywords: + patch
stage: patch review
pull_requests: + pull_request18943
2020-04-19 20:17:57BTaskayasetnosy: + BTaskaya
2020-04-19 20:16:17gvanrossumcreate