This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Py_CompileString no longer allows to tell "incomplete input" from "invalid input"
Type: Stage: resolved
Components: C API Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: iritkatriel, lys.nikolaou, mloskot, pablogsal, python-dev
Priority: normal Keywords: patch

Created on 2022-01-24 15:48 by mloskot, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
Py_CompileString-Py36-vs-Py310.png mloskot, 2022-01-24 15:48 VS 2022 debugging session of the sample program from the FAQ presenting the difference in the Py_CompileString behaviour between Python 3.6 and 3.10 on Windows.
Pull Requests
URL Status Linked Edit
PR 30925 merged python-dev, 2022-01-26 19:33
PR 30933 merged pablogsal, 2022-01-26 23:50
PR 30934 merged pablogsal, 2022-01-26 23:52
Messages (12)
msg411484 - (view) Author: Mateusz Loskot (mloskot) * Date: 2022-01-24 15:48
Something has changed in Python 3.7 through 3.10 (I'm observing it in 3.10) in behaviour of the Python C API function Py_CompileString such that for an incomplete input it no longer raises

SyntaxError: "unexpected EOF while parsing"

but

IndentationError: expected an indented block after ...

The new behaviour makes the sample program from the "How do I tell “incomplete input” from “invalid input”?" at https://docs.python.org/3/faq/extending.html no longer work as described there.

For example:

```
for i in []:
```

raises

IndentationError: expected an indented block after 'for' statement on line 1


```
if True:
```

raises

IndentationError: expected an indented block after 'if' statement on line 1

instead of 

SyntaxError: unexpected EOF while parsing

This effectively makes it impossible to detect incomplete input using the Py_CompileString in applications where it is not possible to use PyRun_InteractiveLoop.

I have failed to identify what could be related changes in the release notes and the documentation does not seem to offer any update on that.
So, I'm assuming the new behaviour is not desired or expected.

Attached, is the VS 2022 screenshot with debugging session of the sample program from the FAQ presenting the difference in the behaviour between Python 3.6 and 3.10 on Windows.
msg411591 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2022-01-25 13:59
The python parser was completely rewritten in version 3.9, so it's likely that this caused changes to error messages.

Adding @pablogsal and @lys.nikolaou in case they can say more.
msg411596 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-25 14:32
Indeed, these new errors are due to the new parser. Unfortunately, the new parser doesn't allow to check against incomplete input the way the old one did so that piece of documentation is unfortunately outdated.

We should remove it to avoid further confusion.
msg411597 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2022-01-25 14:36
Mateusz, would you like to submit a patch to remove this section from the faq?
msg411605 - (view) Author: Mateusz Loskot (mloskot) * Date: 2022-01-25 15:04
Irit, I can and I will submit a patch.

However, I'm very keen to learn what is an alternative solution then to distinguish hard invalid from incomplete input. IOW, what would be the new way of achieving what's described in the old FAQ?

I believe it would be more useful to the community if I updated the sample in the FAQ instead of just removing it.
msg411609 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-25 15:10
> However, I'm very keen to learn what is an alternative solution then to distinguish hard invalid from incomplete input. IOW, what would be the new way of achieving what's described in the old FAQ?

Unfortunately, there is no new way to do this from C due to how the new parser works. The only supported way is to manually import the codeop module from C (https://docs.python.org/3/library/codeop.html#module-codeop) and use its API as IDLE does, but this API can differ and will present its own complications, as unfortunately is implemented in a bit hacky way.
msg411787 - (view) Author: Mateusz Loskot (mloskot) * Date: 2022-01-26 19:34
Not quite the answer I'd like to received, but thank you very much for the explanation.

I've submitted pull request removing the deprecated FAQ entry
https://github.com/python/cpython/pull/30925
msg411811 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-26 23:49
New changeset f0a648152f2d8011f47cc49873438ebaf01d3f82 by Mateusz Łoskot in branch 'main':
bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925)
https://github.com/python/cpython/commit/f0a648152f2d8011f47cc49873438ebaf01d3f82
msg411814 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-27 00:16
New changeset dafada393f9a790461430e2493ea1379e938b51a by Pablo Galindo Salgado in branch '3.9':
[3.9] bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925) (GH-30934)
https://github.com/python/cpython/commit/dafada393f9a790461430e2493ea1379e938b51a
msg411815 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-01-27 00:16
New changeset c7af838805ddf52320bce3d5978bfdd37eed1b3a by Pablo Galindo Salgado in branch '3.10':
[3.10] bpo-46502: Remove "How do I tell incomplete input" from FAQ (GH-30925) (GH-30933)
https://github.com/python/cpython/commit/c7af838805ddf52320bce3d5978bfdd37eed1b3a
msg413251 - (view) Author: Mateusz Loskot (mloskot) * Date: 2022-02-14 19:06
Possibly, the new partial-input mode of the parser (https://bugs.python.org/issue46521#msg412832) will improve this issue in 3.11. I'm going to play with it soon and I will report back.
msg413253 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2022-02-14 19:14
Cool! Be aware that using it outside codeop is currently unsupported:

https://github.com/python/cpython/blob/5d53cf30f9cb3758849e859db5d4602cb7c521f7/Lib/codeop.py#L43-L47

So have in mind that the flag, mechanism and semantics can change without previous notice
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90660
2022-02-14 19:14:23pablogsalsetmessages: + msg413253
2022-02-14 19:06:42mloskotsetmessages: + msg413251
2022-01-27 00:17:04pablogsalsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2022-01-27 00:16:54pablogsalsetmessages: + msg411815
2022-01-27 00:16:53pablogsalsetmessages: + msg411814
2022-01-26 23:52:18pablogsalsetpull_requests: + pull_request29113
2022-01-26 23:50:49pablogsalsetpull_requests: + pull_request29112
2022-01-26 23:49:14pablogsalsetmessages: + msg411811
2022-01-26 19:34:40mloskotsetmessages: + msg411787
2022-01-26 19:33:22python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request29104
stage: patch review
2022-01-25 15:10:18pablogsalsetmessages: + msg411609
2022-01-25 15:04:49mloskotsetmessages: + msg411605
2022-01-25 14:36:46iritkatrielsetversions: + Python 3.9, Python 3.11
2022-01-25 14:36:35iritkatrielsetmessages: + msg411597
2022-01-25 14:32:05pablogsalsetmessages: + msg411596
2022-01-25 13:59:38iritkatrielsetnosy: + iritkatriel, pablogsal, lys.nikolaou
messages: + msg411591
2022-01-24 15:48:56mloskotcreate