This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: REPL shows continuation prompt (...) when comment or space entered
Type: Stage: resolved
Components: Interpreter Core Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Phaqui, gvanrossum, miss-islington, ned.deily, terry.reedy
Priority: low Keywords: patch

Created on 2019-11-03 16:15 by gvanrossum, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17421 merged BTaskaya, 2019-11-30 21:43
PR 17516 merged miss-islington, 2019-12-09 04:38
PR 17522 merged miss-islington, 2019-12-09 11:13
Messages (15)
msg355903 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-03 16:15
This has always bothered me, and it shouldn't be necessary.

This session:

>>> #foo

should really have been

>>> #foo

It's confusing that the REPL prompt switches to "..." here, for no good reason. It should just treat the line as empty.

Ditto if you enter a space (there's an invisible space after the first prompt):

msg356205 - (view) Author: Anders Lorentsen (Phaqui) * Date: 2019-11-07 17:52
As a person without much experience, it sounded like a simple enough task, but having dug a bit, I found it quite complicated. It seems to me that the interpreter loop (in the standard REPL, that you get when you start ./python, blocks for input somewhere inside a massive function called 'parsetok' (in Parser/parsetok.c). Now, I could maybe investigate further, to have it return to the interpreter loop if it reads a comment (or empty line), but I'm afraid to mess up something.

From my understanding, there aren't that many other choices, because parsetok() doesn't return before you finish the statement (in other words, it does not return if you type a comment line or a blank line - it instead waits for more input, as indicated by the '... ').

Am I way off in concluding that this would be a change to the parser?
msg356214 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-07 22:46
Yes, that's likely where the change should be made.

I think if the *first* token encountered is either NL or COMMENT the parse should be abandoned by the tokenizer.
msg356271 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-08 22:55
Entering 'pass' or a completely blank line results in a new primary prompt, at least on Windows. The Windows REPL otherwise prints ... even for effectively blank lines.  IDLE usually prints a new prompt for effectively blank lines.

>>> #a
>>> # a
>>>  #a

I agree that these look better.  This behavior comes from code.InteractiveInterpreter and ultimately codeop.

def _maybe_compile(compiler, source, filename, symbol):
    # Check for source consisting of only blank lines and comments
    for line in source.split("\n"):
        line = line.strip()
        if line and line[0] != '#':
            break               # Leave it alone
        if symbol != "eval":
            source = "pass"     # Replace it with a 'pass' statement

As noted above, 'pass\n' is treated the same as '\n'

The first line above originally had a space, but IDLE appears to strip trailing whitespace also, even outside of comments.  (For an ending '\ ', this prevents SyntaxError, but maybe this is a bad lesson for beginners.)  However, I did find a case with an unnecessary continuation line.

>>>  # a

This puzzles me, as it should be treated exactly the same as without the space after '#'.  ast.dump(ast.parse(' # a\n', '', 'single')) gives the same result, 'Module(body=[], type_ignores=[])', as without.
msg356319 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-10 05:53
Regarding the IDLE mystery, *if* there's a difference between how it treats " # a" and "# a", this must be due to some part of the code that's invoked before _maybe_compile() is called, right?

But that's immaterial to this bug -- I'm only complaining about the "builtin" REPL.
msg356346 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-11 03:24
Fix corner case bugs in IDLE would definitely be a separate issue.  But is the 'fix' in _maybe_compile at all applicable to the REPL?  Or might a parser change REPL fix make the code in _maybe_compile unneeded?
msg356348 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-11 04:11
> But is the 'fix' in _maybe_compile at all applicable to the REPL?  Or might a parser change REPL fix make the code in _maybe_compile unneeded?

I don't know.  Most of the contortions in are meant to emulate what the parser does without help in the REPL: keep asking for input until one of the following happens:

- a syntax error is found;

- a simple statement is parsed till completion;

- an empty line is found at a point where a compound statement is potentially complete.

The bug here is that apparently the above conditions aren't quite enough, and it seems we need to add:

- a line that contains just whitespace and/or a comment is seen before anything else.

The reason that IDLE has to reimplement similar logic is that the parser (actually, the tokenizer) isn't written as a coroutine to which you send lines you read -- it's written as a blocking function that you pass a file and the function will attempt to read lines from the file.  The function returns a parse tree or raise an error.  That model doesn't work in IDLE, which needs to stay reactive while the shell window is in the middle of a statement.

I think we'll find that the bug is *very* old.  IIRC the initial releases of Python didn't have the rule that indentation is ignored between matching parentheses/brackets/braces.  In those days the tokenizer also didn't gracefully skip blank lines, or lines with comments that weren't aligned with the current indentation level.  So then checking for empty lines was good enough.
msg358051 - (view) Author: miss-islington (miss-islington) Date: 2019-12-09 04:36
New changeset 109fc2792a490ee5cd8a423e17d415fbdedec5c8 by Miss Islington (bot) (Batuhan Taşkaya) in branch 'master':
bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421)
msg358052 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-12-09 04:37
I'd like to backport this to 3.8.1 at least. Are people interested in getting it backported to earlier versions?
msg358053 - (view) Author: miss-islington (miss-islington) Date: 2019-12-09 04:56
New changeset 184a3812b81e2f7d4bc6453bf7ceabe8ac590202 by Miss Islington (bot) in branch '3.8':
bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421)
msg358054 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-12-09 05:07
So 3.8.1 got backported by Miss Islington. Do we want this in earlier releases?
msg358055 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-12-09 06:34
I would prefer that.  I think treating '\n' and ' \n' differently is a bit of a bug.  And the fix pretty well matches code/codeop behavior.  I have so far not imagined how it could break code.    But you could let Ned Deily decide, before the next rc, if you want.

I am neutral on 2.7.
msg358071 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-09 11:38
New changeset 188d5ae6f047342f3d6860646ccf1a523ef8b0ed by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-38673: dont switch to ps2 if the line starts with comment or whitespace (GH-17421) (GH-17522)
msg358107 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-12-09 14:54
Ned agreed, it's merged into 3.7, so let's close.
msg358110 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-12-09 15:12
(And I'm giving up on 3.6 and 2.7 as these are close to their end of life.)
Date User Action Args
2022-04-11 14:59:22adminsetgithub: 82854
2019-12-09 15:12:47gvanrossumsetmessages: + msg358110
2019-12-09 14:54:58gvanrossumsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-12-09 14:54:43gvanrossumsetmessages: + msg358107
2019-12-09 11:38:41ned.deilysetnosy: + ned.deily
messages: + msg358071
2019-12-09 11:13:13miss-islingtonsetpull_requests: + pull_request17000
2019-12-09 06:34:29terry.reedysetmessages: + msg358055
versions: - Python 3.5, Python 3.6
2019-12-09 05:07:06gvanrossumsetmessages: + msg358054
2019-12-09 04:56:25miss-islingtonsetmessages: + msg358053
2019-12-09 04:38:16miss-islingtonsetpull_requests: + pull_request16993
2019-12-09 04:37:39gvanrossumsetmessages: + msg358052
2019-12-09 04:36:39miss-islingtonsetnosy: + miss-islington
messages: + msg358051
2019-11-30 21:43:10BTaskayasetkeywords: + patch
stage: patch review
pull_requests: + pull_request16902
2019-11-11 04:11:11gvanrossumsetmessages: + msg356348
2019-11-11 03:24:48terry.reedysetmessages: + msg356346
2019-11-10 05:53:43gvanrossumsetmessages: + msg356319
2019-11-08 22:55:09terry.reedysetnosy: + terry.reedy
messages: + msg356271
2019-11-07 22:46:49gvanrossumsetmessages: + msg356214
2019-11-07 17:52:55Phaquisetnosy: + Phaqui
messages: + msg356205
2019-11-03 16:15:35gvanrossumcreate