classification
Title: IDLE: Fix shell comment anomalies
Type: behavior Stage: test needed
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: taleinat, terry.reedy
Priority: normal Keywords:

Created on 2021-06-06 20:44 by terry.reedy, last changed 2021-06-07 10:44 by taleinat.

Messages (5)
msg395214 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-06-06 20:44
Spinoff from #38673, about standard REPL, msg356271 (me) and msg356348 (Guido).  In the following interactions, no blank lines were entered.

3.9 behavior
>>> #a
>>> # a
>>>  #a
>>>  # a
 
>>> 
Mystery 1: why the blank continuation line?

I previously wrote
"ast.dump(ast.parse(' # a\n', '', 'single')) gives the same result, 'Module(body=[], type_ignores=[])', as without [space after #]".
Today, 3.8.10, 3.9.5, 3.10, and 3.11 say "unexpected EOF while parsing".

3.10 behavior
>>> #a
...  
>>> # a
>>>  #a
>>>  # a
...  
>>>
Mystery 2: why the new continuation line after '#a'?

3.11 behavior
>>> #a
>>> # a
>>> #a
>>>  #a
>>>  # a
...  
>>>
Mystery 3: why does the 3.10 regression for '#a' disappear?

Perhaps IDLE should handle initial blank lines itself, but I will investigate what codeop._maybe_compile is getting and doing in the different cases first.
msg395227 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-06-06 22:48
I added debug prints to _maybe_compile and confirmed

1) trailing whitespace (' ' and '\t' at least) is removed before this function is called.  I presume in IDLE rather than code.II, but cannot find where.  It is not with .rstrip.  (Note: doing so after '\' is a bug in that it lets buggy input such as 'a\ \n  + 2' run instead of raising.)

2. code is otherwise delivered intact and blanks lines become 'pass'.  Thus any difference noted above is not due to compile().

The report was based on Windows with 3.9.5, 3.10.0b2, and fresh main.  I repeated the Shell experiments on a Mac Airbook, and a slower machine, with 3.10.b1.  There was never a spurious ... -- once the proper >>> was printed.  However, I sometimes saw ... appear very briefly, only to be overwritten with >>>.  (I saw this once, *very briefly*, on Windows with main, on the first comment I tried.)  I also saw ... appear and disappear when there was a SyntaxError or print output.  I suspect that Sidebar always adds ..., only to be deleted or overwritten when it is a mistake.

I then tried 3.9.5 on the Mac and saw a spurious blank line with ' # a' once, on the first try, but not again in several attempts.  There seems to be a 'warmup' effect.

My conclusion so far: sidebar might be a culprit, but because of the 3.9 behavior, I think it more likely a victim of pyshell prematurely marking the new line as a possible continuation line.  As long as there was not continuation prompt, this might seem innocuous.  But it might also be a factor in other spurious blank lines.  In particular,

>>> if 1: # Hit return on indented line.
...     print(2)
... 
...     < undeleted 4 space indent
    2
>>> if 1: # Delete indent first.  Get proper behavior.
...     print(2)
...
    2
>>>

I am stopping here.  Tal, what do you think with your better knowledge of pyshell internals?
msg395258 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-06-07 10:39
> 1) trailing whitespace (' ' and '\t' at least) is removed before this function is called.  I presume in IDLE rather than code.II, but cannot find where.  It is not with .rstrip.

You're probably looking for this code in EditorWindow.newline_and_indent_event():

# Strip whitespace after insert point.
while text.get("insert") in " \t":
    text.delete("insert")

See: https://github.com/python/cpython/blob/89e50ab36fac6a0e7f1998501f36fcd2872a6604/Lib/idlelib/editor.py#L1390
msg395259 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-06-07 10:41
> However, I sometimes saw ... appear very briefly, only to be overwritten with >>>.

This is a known limitation of the current sidebar implementation, which was very difficult to avoid and was considered minor enough to let be for now.
msg395260 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2021-06-07 10:44
The sidebar doesn't seem to be causing this issue, it's just making it a bit more visible, since what was previously a blank line now also has a more visible "..." continuation prompt.
History
Date User Action Args
2021-06-07 10:44:58taleinatsetmessages: + msg395260
2021-06-07 10:41:31taleinatsetmessages: + msg395259
2021-06-07 10:39:52taleinatsetmessages: + msg395258
2021-06-06 22:48:18terry.reedysetmessages: + msg395227
2021-06-06 20:44:50terry.reedycreate