classification
Title: codeop prematurely raises on 2nd line when closer is needed
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: aroberge, hsfzxjy, pablogsal, terry.reedy
Priority: normal Keywords: patch

Created on 2021-02-08 14:42 by hsfzxjy, last changed 2021-02-11 21:39 by pablogsal. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24483 merged pablogsal, 2021-02-08 21:18
Messages (7)
msg386630 - (view) Author: Xie Jingyi (hsfzxjy) Date: 2021-02-08 14:46
In Python3.10.0a5, `code.interact()` will raise unexpected SyntaxError even if there should be more code.

```
Python 3.10.0a5 (default, Feb  7 2021, 20:14:10) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import code
>>> code.interact()
Python 3.10.0a5 (default, Feb  7 2021, 20:14:10) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> print([1,
... 2,
  File "<console>", line 1
    print([1,
          ^
SyntaxError: '[' was never closed
>>> 
```

Similar things also happen in IDLE.
msg386651 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-08 20:11
IDLE is a Python program that wraps the Python interpreter.  Before blaming IDLE for how code is executed, one should first execute the same code directly in Python.  The example above shows that one should also try code.interact, especially for compilaition issues. (Xie, thank you for this reminder.) IDLE and code.interact use the same code and codeop infrastructure. Most relevant here is codeop.CommandCompiler, which should act the same as codeop.compile_command in the absence of __future__ imports.

The codeop functions, including _maybe_compile, are intended to imitate in python the C REPL code that decides whether to execute, get more input, or raise.  It doesn't always.  (I think it would be good if a C expert compared it to the *current* C code.)

>>> CC = codeop.CommandCompiler
>>> cc("print([1,")  # Returns None, meaning, 'Get more input'.
>>> cc("print([1,\n2,")  
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "F:\dev\3x\lib\codeop.py", line 132, in compile_command
    return _maybe_compile(_compile, source, filename, symbol)
  File "F:\dev\3x\lib\codeop.py", line 106, in _maybe_compile
    raise err1
  File "F:\dev\3x\lib\codeop.py", line 93, in _maybe_compile
    code1 = compiler(source + "\n", filename, symbol)
  File "F:\dev\3x\lib\codeop.py", line 111, in _compile
    return compile(source, filename, symbol, PyCF_DONT_IMPLY_DEDENT)
  File "<input>", line 1
    print([1,
          ^
SyntaxError: '[' was never closed

If the opening '[' is removed, then ']' is replaced with (.  If '[' is replaced with '{' and the items adjusted, ']' is replaced with '}'.  In all three cases, CC should return None to ask for more, as the REPL does.
msg386659 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-08 20:30
test.test_codeop's test_incomplete is incomplete in that it tests several missing-closer single lines, like the above and "[x for x in (", all of which pass, but no corresponding multiple line snippets, like the above. The only multiple line snippets are for compound statements.  As far as I could see, the same is true for test_valid.

Are there similar tests for the REPL?  Are they more complete?  Should not both get the same tests?
msg386665 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-02-08 20:59
> (I think it would be good if a C expert compared it to the *current* C code.)

That would be very beneficial but is quite a big task, specially given that codeop design is quite surprising. I personally find quite confusing the way codeop is handling the decisions on continuing or not (is based on compiling the code with different levels of new lines and then checking the repr() of the exceptions). It would be good if someone familiar with "codeop" design can take a look on what's the best way to proceed.
msg386743 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-02-09 20:08
New changeset dbb228189b4eb7ab41f326eb79dae669b2c81177 by Pablo Galindo in branch 'master':
bpo-43163: Handle unclosed parentheses in codeop (GH-24483)
https://github.com/python/cpython/commit/dbb228189b4eb7ab41f326eb79dae669b2c81177
msg386834 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-11 20:49
This issue is for 3.10 only since the never-closed message was only added for 3.10.0a5 and not backported.

Pablo, Is there any sane possibility that the compiler could, at least sometimes, raise an IncompleteSyntax subclass of SyntaxError when code is merely incomplete, as opposed to wrong?

Do you happen to know where the REPL code is?  (If not, I will ask elsewhere.)
msg386837 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-02-11 21:39
>Pablo, Is there any sane possibility that the compiler could, at least sometimes, raise an IncompleteSyntax subclass of SyntaxError when code is merely incomplete, as opposed to wrong?

I'm assuming you mean the parser (when the compiler runs the code must have been correct or otherwise the parser would have raised before).

In case of the parser, I'm not sure but the answer is generally no. The parser doesn't have a concept of "partial Syntax" so that should be defined. There is also the problem that some of these syntax error come from the tokenizer which has its own rules. We could probably do more in the tokenizer than in the parser but this would add considerable complexity in the general case.
History
Date User Action Args
2021-02-11 21:39:50pablogsalsetmessages: + msg386837
2021-02-11 20:49:39terry.reedysetmessages: + msg386834
2021-02-09 20:08:14pablogsalsetmessages: + msg386743
2021-02-09 20:07:54pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-02-08 21:18:14pablogsalsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request23275
2021-02-08 20:59:52pablogsalsetmessages: + msg386665
2021-02-08 20:33:51terry.reedysettitle: codeop prematurely raises SyntaxError when closer is needed -> codeop prematurely raises on 2nd line when closer is needed
2021-02-08 20:30:51terry.reedysetmessages: + msg386659
stage: needs patch
2021-02-08 20:11:19terry.reedysettitle: codeop prematurely raises SyntaxError when ']' is needed -> codeop prematurely raises SyntaxError when closer is needed
2021-02-08 20:11:05terry.reedysettitle: code.interact() unexpectedly raises exception when there may be more code -> codeop prematurely raises SyntaxError when ']' is needed
nosy: + pablogsal

messages: + msg386651

components: - IDLE
2021-02-08 15:27:33arobergesetnosy: + aroberge
2021-02-08 15:08:40hsfzxjysettitle: code.interact() unexpected raises exception when there may be more code -> code.interact() unexpectedly raises exception when there may be more code
2021-02-08 14:47:31hsfzxjysetassignee: terry.reedy

components: + IDLE, Library (Lib)
nosy: + terry.reedy
2021-02-08 14:46:07hsfzxjysetmessages: + msg386630
2021-02-08 14:42:40hsfzxjycreate