classification
Title: PEG Parser: Cannot used starred expression in parenthesised expr
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lys.nikolaou Nosy List: gousaiyang, gvanrossum, lys.nikolaou, miss-islington, pablogsal, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-05-15 12:04 by lys.nikolaou, last changed 2021-01-03 02:01 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24014 closed pablogsal, 2020-12-30 21:15
PR 24015 closed pablogsal, 2020-12-30 21:23
PR 24019 closed pablogsal, 2020-12-31 03:10
PR 24027 merged lys.nikolaou, 2020-12-31 15:34
PR 24068 merged lys.nikolaou, 2021-01-03 00:38
Messages (12)
msg368936 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-15 12:04
The new PEG parser fails when a parenthesised expression with a single child (a group) contains a starred expression. Example:

╰─ ./python.exe
Python 3.9.0a6+ (heads/master-dirty:4a12d12186, May 15 2020, 14:53:45)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.parse('(*a)')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lysnikolaou/Repositories/cpython/Lib/ast.py", line 50, in parse
    return compile(source, filename, mode, flags,
  File "<unknown>", line 1
    (*a)
        ^
SyntaxError: invalid syntax

This was valid syntax up until now:

╰─ ./python.exe -X oldparser
Python 3.9.0a6+ (heads/master-dirty:4a12d12186, May 15 2020, 14:53:45)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ast
>>> ast.dump(ast.parse('(*a)'))
"Module(body=[Expr(value=Starred(value=Name(id='a', ctx=Load()), ctx=Load()))], type_ignores=[])"
msg368938 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-05-15 12:13
Whoops, false alarm.

It's just that we moved the check for invalid starred expressions to the parser, while it previously was in the compiler.

╰─ ./python.exe -X oldparser
Python 3.9.0a6+ (heads/master-dirty:003708bcf8, May 15 2020, 15:08:21)
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> (*a)
  File "<stdin>", line 1
SyntaxError: can't use starred expression here

Sorry for the noise!
msg384086 - (view) Author: Saiyang Gou (gousaiyang) * Date: 2020-12-30 20:29
Well, there is actually a bug:

f1b4a742d8fc">root@f1b4a742d8fc:/# python3.9
Python 3.9.1 (default, Dec  8 2020, 03:24:52)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = [1, 2]
>>> y = 3
>>> *x, y
(1, 2, 3)
>>> (*x), y
  File "<stdin>", line 1
    (*x), y
     ^
SyntaxError: can't use starred expression here

f1b4a742d8fc">root@f1b4a742d8fc:/# python3.8
Python 3.8.6 (default, Oct  6 2020, 03:22:36)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> x = [1, 2]
>>> y = 3
>>> *x, y
(1, 2, 3)
>>> (*x), y
(1, 2, 3)

This is different from the previous message where the starred expression is "alone" (and thus invalid). Since this bug happens in 3.9 but not in 3.8, it might be due to the PEG parser.

Also,
f1b4a742d8fc">root@f1b4a742d8fc:/# python3.9
Python 3.9.1 (default, Dec  8 2020, 03:24:52)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> del *x
  File "<stdin>", line 1
    del *x
        ^
SyntaxError: cannot delete starred
>>> del (*x)
  File "<stdin>", line 1
    del (*x)
         ^
SyntaxError: can't use starred expression here

The latter case should also report "SyntaxError: cannot delete starred".
msg384089 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-30 22:30
Honestly this seems like a bug in 3.8 to me (if it indeed behaves like this):

>>> (*x), y
(1, 2, 3)

Every time I mistakenly tried (*x) I really meant (*x,), so it's surprising that (*x), y would be interpreted as (*x, y) rather than flagging (*x) as an error.

Please don't "fix" this even if it is a regression.
msg384091 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-12-30 23:14
It makes sense to me to be able to do `(*a), b` if I can do `*a, b`, but I don't really have a strong opinion on it.
msg384092 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-12-30 23:22
Parenthesis can be added around expression. But `*a` is not an expression (as well as `+`, `or`, `1:5`, you cannot surround them with parenthesis).
msg384093 - (view) Author: Saiyang Gou (gousaiyang) * Date: 2020-12-30 23:39
Also the current behavior allows `(*x), y = 1` assignment. If `(*x)` is to be totally disallowed, `(*x), y = 1` should also be rejected.
msg384094 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-30 23:41
> It makes sense to me to be able to do `(*a), b` if I can do `*a, b`, but I don't really have a strong opinion on it.

I disagree. *a is not an expression, so the normal rules for parenthesizing those don't apply. I've always thought of *a as a feature of the "comma" syntax. Note too that (**a) is not valid and never was. Also note that 2.7 doesn't support f((*a)). In fact 3.4 doesn't either -- but 3.5 does.

I don't know how this slipped into earlier Python 3 versions -- apparently there aren't tests for this, and it's not used in popular 3rd code either, or we would have found out when we first implemented PEP 617.

Most likely it's due to the general problem where the parser would just accept parenthesized stuff in various places where it shouldn't (e.g. also f(a=1) could be spelled as f((a)=1) -- this was fixed in 3.8).

> Also the current behavior allows `(*x), y = 1` assignment. If `(*x)` is to be totally disallowed, `(*x), y = 1` should also be rejected.

I agree.
msg384096 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2020-12-31 00:27
Yup, this all sounds much more reasonable. Thanks for the explanation, Guido.

> Also the current behavior allows `(*x), y = 1` assignment. If `(*x)` is to be totally disallowed, `(*x), y = 1` should also be rejected.

This is allowed in 3.9.1 and 3.10.0a2, but not allowed in 3.9.0 and 3.10.0a1. I'll work on finding out when this got messed up and fix it.
msg384098 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-12-31 02:04
> I'll work on finding out when this got messed up and fix it.

Apparently is 

commit bca701403253379409dece03053dbd739c0bd059 (HEAD)
Author: Lysandros Nikolaou <lisandrosnik@gmail.com>
Date:   Tue Oct 27 00:42:04 2020 +0200

    bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)


~/github/python/master v3.10.0a2~61 9s
❯ ./python -c "(*x), y = 1,2"

~/github/python/master v3.10.0a2~61
❯ git checkout HEAD^
Previous HEAD position was bca7014032 bpo-42123: Run the parser two times and only enable invalid rules on the second run (GH-22111)
HEAD is now at c8c4200b65 bpo-42157: Convert unicodedata.UCD to heap type (GH-22991)

~/github/python/master remotes/welikeparsers/master
❯ make -j -s
 CC='gcc -pthread' LDSHARED='gcc -pthread -shared    ' OPT='-g -Og -Wall'       _TCLTK_INCLUDES='' _TCLTK_LIBS=''       ./python -E ./setup.py -q build

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc                  atexit                pwd
time


~/github/python/master remotes/welikeparsers/master
❯ ./python -c "(*x), y = 1,2"
  File "<string>", line 1
    (*x), y = 1,2
     ^
SyntaxError: can't use starred expression here
msg384244 - (view) Author: Lysandros Nikolaou (lys.nikolaou) * (Python committer) Date: 2021-01-02 23:14
New changeset 2ea320dddd553298038bb7d6789e50e199332f66 by Lysandros Nikolaou in branch 'master':
bpo-40631: Disallow single parenthesized star target (GH-24027)
https://github.com/python/cpython/commit/2ea320dddd553298038bb7d6789e50e199332f66
msg384247 - (view) Author: miss-islington (miss-islington) Date: 2021-01-03 00:59
New changeset 9a608ac17c284008d3c2986a4a8b194f84488e56 by Lysandros Nikolaou in branch '3.9':
[3.9] bpo-40631: Disallow single parenthesized star target (GH-24027) (GH-24068)
https://github.com/python/cpython/commit/9a608ac17c284008d3c2986a4a8b194f84488e56
History
Date User Action Args
2021-01-03 02:01:34pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-01-03 00:59:56miss-islingtonsetnosy: + miss-islington
messages: + msg384247
2021-01-03 00:38:00lys.nikolaousetpull_requests: + pull_request22901
2021-01-02 23:14:28lys.nikolaousetmessages: + msg384244
2020-12-31 15:34:15lys.nikolaousetpull_requests: + pull_request22867
2020-12-31 03:10:25pablogsalsetpull_requests: + pull_request22859
2020-12-31 02:04:42pablogsalsetmessages: + msg384098
2020-12-31 00:27:32lys.nikolaousetmessages: + msg384096
2020-12-30 23:41:18gvanrossumsetmessages: + msg384094
2020-12-30 23:39:22gousaiyangsetmessages: + msg384093
2020-12-30 23:22:48serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg384092
2020-12-30 23:14:40lys.nikolaousetmessages: + msg384091
2020-12-30 22:30:20gvanrossumsetmessages: + msg384089
2020-12-30 21:23:41pablogsalsetpull_requests: + pull_request22855
2020-12-30 21:15:10pablogsalsetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request22854
2020-12-30 20:37:24pablogsalsetstatus: closed -> open
resolution: not a bug -> (no value)
2020-12-30 20:29:36gousaiyangsetnosy: + gousaiyang
messages: + msg384086
2020-05-15 12:14:34lys.nikolaousetstatus: open -> closed
resolution: not a bug
stage: resolved
2020-05-15 12:13:59lys.nikolaousetmessages: + msg368938
2020-05-15 12:04:27lys.nikolaoucreate