classification
Title: compile_command exec not raising syntax error with new PGEN Parser
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, mbussonn, pablogsal
Priority: normal Keywords: patch

Created on 2020-05-10 23:03 by mbussonn, last changed 2020-05-11 03:40 by gvanrossum. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 20030 merged pablogsal, 2020-05-10 23:20
Messages (3)
msg368594 - (view) Author: Matthias Bussonnier (mbussonn) * Date: 2020-05-10 23:03
compile_command  used to produce syntax error in exec mode:

```
$ python -c "import codeop; codeop.compile_command('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...python3.8/codeop.py", line 125, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "...python3.8/codeop.py", line 100, in _maybe_compile
    raise err1
  File "...python3.8/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "...python3.8/codeop.py", line 105, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
```

This happens to not be the case anymore in master, where it simply return `None` for above invalid input. 

```
$ python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"
```

or many other:

```
 $ python
Python 3.9.0a6+ (heads/master:2cc9b8486d, May 10 2020, 15:52:00)
[Clang 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import codeop;
>>> codeop.compile_command('def a-b', symbol='exec')
>>> codeop.compile_command('await?', symbol='exec')
>>> codeop.compile_command('=!=', symbol='exec')
>>> codeop.compile_command('a await raise b', symbol='exec')
>>> codeop.compile_command('a await raise b?+1', symbol='exec')
```


It is problematic as this is used in many places to decide whether code is valid, and for example in IPython to know wether we should insert a new line, or try to execute the code. 

It seem to be due to the new PGEN parser as setting PYTHONOLDPARSER solve the issue:

```
$ PYTHONOLDPARSER=1 python -c "import codeop; codeop.CommandCompiler()('raise = 2', symbol='exec')"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 171, in __call__
    return _maybe_compile(self.compiler, source, filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 100, in _maybe_compile
    raise err1
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 87, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "/Users/bussonniermatthias/dev/cpython/Lib/codeop.py", line 136, in __call__
    codeob = compile(source, filename, symbol, self.flags, True)
  File "<input>", line 1
    raise = 2
          ^
SyntaxError: invalid syntax
```

`single` and `eval` appear to behave fine.
msg368597 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 23:21
Let's analyze this here rather than in issue40334 (which is getting top-heavy :-).
msg368600 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-05-11 00:41
New changeset 5b956ca42de37c761562e9c9aeb96a0e67606e33 by Pablo Galindo in branch 'master':
bpo-40585: Normalize errors messages in codeop when comparing them (GH-20030)
https://github.com/python/cpython/commit/5b956ca42de37c761562e9c9aeb96a0e67606e33
History
Date User Action Args
2020-05-11 03:40:31gvanrossumsetstage: patch review -> resolved
2020-05-11 03:20:21gvanrossumsetstage: resolved -> patch review
2020-05-11 00:41:39pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-05-11 00:41:36pablogsalsetmessages: + msg368600
2020-05-10 23:21:17gvanrossumsetnosy: + gvanrossum
messages: + msg368597
2020-05-10 23:20:36pablogsalsetkeywords: + patch
nosy: + pablogsal

pull_requests: + pull_request19340
stage: patch review
2020-05-10 23:03:34mbussonncreate