IDLE inserts an extra blank line after the prompt after encountering a SyntaxError:

>>> 1 + 2
3
>>> print('Hello')
Hello
                                     v-- Missing single quote!
>>> d = {1: 'uno', 2: 'dos', 3: 'tres}
	 
SyntaxError: EOL while scanning string literal
>>> print('Hello')
	 <-- Extra blank line and whitespace (tab and space).
Hello
>>> 1 + 2
	 <-- Extra blank line and whitespace (tab and space).
3
>>> 

Notice the line starting with ">>> d =" above contains a missing single quote which causes a "SyntaxError: EOL while scanning string literal". This causes IDLE to insert extra blank lines with one tab and one space after every input.

The old behavior looked like:

>>> 1 + 2
3
>>> print('Hello')
Hello
>>> d = {1: 'uno', 2: 'dos', 3: 'tres}
     
SyntaxError: EOL while scanning string literal
>>> print('Hello')
Hello
>>> 1 + 2
3

The SyntaxError is not relevant. Interactive CPython:

>> d=[]
>> d=()
>> d={}
>>

Erroneous extra lines in IDLE 3.6+ Shell but not editor:

>>> d = []
	 
>>> d=()
	 
>>> d={}
	 
>>> d=[i for i in [1]]
	 
>>> 

The 'blank' lines are indents produced by IDLE's smart indent mechanism, which is trigger by keying '\n', *before* the code is tentatively compiled.

While the extra lines are an error for the examples above, they are arguably correct for your example, where there is no closing '}'. The indenter treats it the same as if there were a closing quote, as in the following, which *is* the same in shell and editor, and correct.

d = {1: 'one}'
     # Indent lines up next dict item with the one above.

Even though your example is no a bug, it lead me to discover a regression in current 3.6+. In the past year, there have been a couple of patches that touched the autoindent code.

def funcname(param = 'somestring)
             # Indent for next param.

is another situation where \n is treating as closing '.

Of late, I've also encountered two recurring problems along these lines. At some point in the IDLE session, the auto-indents become large and this survives a restart-shell. Another problem is that an extra \n is emitted after every result which gives an odd double-spaced effect.

When added to other existing bugs, this has greatly impaired the usability of IDLE for teaching (the tool tips no longer display on the newest mac builds and periodically the mouse pointer is mistargeted by 1/4 of the screen until the screen is resized, and periodically I encounter cases where pressing any key in the interactive session results in pasting the current contents of the keyboard on the current entry line).

I did several more experiments to reproduce the reported problems and narrow down when they occur. (Where I say 3.6.3 below, I actually tested 2.7.15 and/or 3.5.4 and assume same behavior until fix for 3.6.4.)

>>> a=1
>>> a
1  # no extra space
>>> d={1:'a',
       }  # 3 space autoindent in 3.6.3-, 7 in 3.6.4+ after fix.
>>> ===  # 2nd '=' marked with red 'error' background
SyntaxError: invalid syntax  # no extra space for this SyntaxError
>>> a
1  # no extra space
>>> d={1:'a
       |<-- 3 or 7 space indent in 3.6.3-, 3.6.4+
SyntaxError: EOL while scanning string literal
>>> a
       |<-- Regression: persistent only after fix
1
>>> if a    : d={1:'a
		|<-- new indent, 2 tabs + space, correct
SyntaxError: EOL while scanning string literal
>>> a
		|<-- extra line with larger indent.
1
>>> if a              : e{3:===}
		|<-- indent not increased with matched {}.
SyntaxError: invalid syntax

Fresh start:
>>> a=1
>>> d==={  # error before unmatched {
	|<-- indent
SyntaxError: invalid syntax
>>> a
1

Conclusion: IDLE has had a buglet in adding an indented blank line before 'SyntaxError' when the erroneous line (or maybe statement) has an unmatched opener. Before 3.6.4, indents after unmatched openers in the first line of a statement were wrong because the prompt was ignored. However, the fix introduced a regression in making the corrected indent persistent. I will have to check whether the persistence is from an uninitialized value or from erroneously including prior statements in the calculation.

If possible, check syntax first and only request a smart indent when the statement is correct but incomplete. This should fix the buglet and might nullify the persistence, though I would like to fix the persistence also.
---

I have occasionally experienced unrequested pasting of previous input or output after the 1st prompt after a restart, but it is so rare for me that I have not yet detected a pattern.

bpo-35467 is about unwanted auto-pasting.

This code in editor.py controls the text that is parsed for smart indenting:
if not self.context_use_ps1:
for context in self.num_context_lines:
startat = max(lno - context, 1)
startatindex = repr(startat) + ".0"
rawtext = text.get(startatindex, "insert")
y.set_code(rawtext)
bod = y.find_good_parse_start(
self.context_use_ps1, self._build_char_in_string_func(startatindex))
if bod is not None or startat == 1:
break
y.set_lo(bod or 0)
else:
r = text.tag_prevrange("console", "insert")
if r:
startatindex = r[1]
else:
startatindex = "1.0"
rawtext = text.get(startatindex, "insert")
y.set_code(rawtext)
y.set_lo(0)

The if not self.context_use_ps1 basically says whether the window is an editor or shell. At a high level, the editor code goes back a certain number of lines from the current cursor and the shell window goes back to just the current statement.

bpo-31858 improved the use of sys.ps1 (the prompt) and it removed setting self.context_use_ps1 in pyshell. This meant that the else above was never accessed and that the shell was parsing all the text, not just the current statement.

bpo-31858 introduced self.prompt_last_line that is set to '' in the editor and set to the prompt in the shell. Replacing self.context_use_ps1 with self.prompt_last_line allows the else above to be called.

bpo-32989 addresses a bug discovered with adding tests to pyparse where the find_good_parse_start call above was actually sending incorrect parameters, so removing context_use_ps1 from that line is not an issue, but rather another bug fix. It might be preferable to merge bpo-32989 first, therefore I'm listing that as a dependency.

I changed the title back to the symptom description, rather than the wrong fix. As Cheryl noted, the cause of the specific regression is the erroneous deletion of one line in pyshell.PyShell. The simplest fix is to put is back.

Globally replacing context_use_ps1 is a different issue, and there is more than one possibility. I will open a new issue and move PR 11307 there.

The reversion also reverts the 'fixes' that came with the error.  Fixing the shell branch, now that is is used, will be a different issue.
>>> d = {1:3,
	 # correct indent, 3.7

>>> d = {1:3,
     # indent not accounting for prompt, after reversion

New changeset 4bc2467 by Terry Jan Reedy in branch 'master':
bpo-34055: Revert deletion of line in IDLE's PyShell (bpo-11346)
4bc2467

New changeset 95dc457 by Miss Islington (bot) in branch '3.7':
bpo-34055: Revert deletion of line in IDLE's PyShell (GH-11346)
95dc457

Even before this fix, in 3.7.2, I no longer see the extra blank lines I reported in msg321135 either on Windows or macOS. I did still see the persistent and expanding indents reported in the opening post. I believe that this is what Raymond also referred to. The one I could reproduce are now gone. Hence, I close this.

Remaining newline and indent issues should be handled in other issues.

• I intend to review the initial PR for bpo-32989 soon. I don't know what its effect might be on the user-visible result.

• Too short indents for open fences needs a new issue.

• Blank line before SyntaxError: interactive python does this.

>>> d={a:'a}
  File "<stdin>", line 1
    d={a:'a}
           ^
SyntaxError: EOL while scanning string literal
>>>

The ^ line is nearly blank. IDLE omits the line echo and ^ and adds a red highlight instead ... + an invisible indent on the next line (now too short). (It also color-codes the error message.)

>>> d={a:'a}<---red highlight to end of line...
   
SyntaxError: EOL while scanning string literal
>>>

Checking syntax first and skipping indent on error would give the following. I don't know how easy this would be.

>>> d={a:'a}<---red highlight to end of line...
SyntaxError: EOL while scanning string literal
>>>

I 'moved' pr-11307 to bpo-35610 by changing the title.

This issue was closed but I still see the problem in 3.7.2. Here's a snippet with line numbers from IDLE:

01 Python 3.7.2 (default, Dec 30 2018, 08:59:00) 
02 [Clang 9.1.0 (clang-902.0.39.2)] on darwin
03 Type "help", "copyright", "credits" or "license()" for more information.
04 >>> 1 + 2
05 3
06 >>> print('Hello')
07 Hello
08 >>> d = {1: 'uno', 2: 'dos', 3: 'tres}
09 	 
10 SyntaxError: EOL while scanning string literal
11 >>> 1 + 2
12 	 
13 3
14 >>> 

Notice that IDLE is inserting an extra blank line at (12) above.

And here's a snippet with line numbers from the Python shell:

01 Python 3.7.2 (default, Dec 30 2018, 08:59:00) 
02 [Clang 9.1.0 (clang-902.0.39.2)] on darwin
03 Type "help", "copyright", "credits" or "license" for more information.
04 >>> 1 + 2
05 3
06 >>> print('Hello')
07 Hello
08 >>> d = {1: 'uno', 2: 'dos', 3: 'tres}
09   File "<stdin>", line 1
10     d = {1: 'uno', 2: 'dos', 3: 'tres}
11                                      ^
12 SyntaxError: EOL while scanning string literal
13 >>> 1 + 2
14 3
15 >>> 

Between lines (13) and (14) there is no extra blank line.

I'm sorry if my initial post was unclear. But the extra blank line is the bug I'm describing. I don't think there should be an extra blank line between (11) and (13) in the IDLE shell. This blank line persists for every input, even after restarts.

I'm on macOS.

I would be interested in debugging the issue locally but I ran into a couple issues trying to do so. When I check out the CPython sources and build the python.exe executable, I get this error when trying to execute IDLE:

$ ./python.exe -m pdb -m idlelib.idle
> /Users/grantj/repos/cpython/Lib/idlelib/idle.py(1)<module>()
-> import os.path
(Pdb) c
Traceback (most recent call last):
  File "/Users/grantj/repos/cpython/Lib/pdb.py", line 1695, in main
    pdb._runmodule(mainpyfile)
  File "/Users/grantj/repos/cpython/Lib/pdb.py", line 1540, in _runmodule
    self.run(code)
  File "/Users/grantj/repos/cpython/Lib/bdb.py", line 585, in run
    exec(cmd, globals, locals)
  File "/Users/grantj/repos/cpython/Lib/idlelib/idle.py", line 1, in <module>
    import os.path
  File "/Users/grantj/repos/cpython/Lib/idlelib/pyshell.py", line 1507, in main
    macosx.setupApp(root, flist)
  File "/Users/grantj/repos/cpython/Lib/idlelib/macosx.py", line 280, in setupApp
    overrideRootMenu(root, flist)
  File "/Users/grantj/repos/cpython/Lib/idlelib/macosx.py", line 181, in overrideRootMenu
    del mainmenu.menudefs[-2][1][0]
IndexError: list assignment index out of range

If I comment out line 181 in /Users/grantj/repos/cpython/Lib/idlelib/macosx.py then I can get IDLE to start. But it will later crash trying to display the first tooltip:

$ ./python.exe -m pdb -m idlelib.idle
> /Users/grantj/repos/cpython/Lib/idlelib/idle.py(1)<module>()
-> import os.path
(Pdb) c
2019-01-02 15:52:57.582 python.exe[23803:6374992] *** Assertion failure in -[_NSCGSWindow setFrame:], /BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1561.60.100/CGS.subproj/NSCGSWindow.m:1002
2019-01-02 15:52:57.588 python.exe[23803:6374992] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid parameter not satisfying: CGRectContainsRect(CGRectMake((CGFloat)INT_MIN, (CGFloat)INT_MIN, (CGFloat)INT_MAX - (CGFloat)INT_MIN, (CGFloat)INT_MAX - (CGFloat)INT_MIN), frame)'
*** First throw call stack:
(
	0   CoreFoundation                      0x00007fff48fa923b __exceptionPreprocess + 171
	1   libobjc.A.dylib                     0x00007fff7023ac76 objc_exception_throw + 48
	2   CoreFoundation                      0x00007fff48faefd2 +[NSException raise:format:arguments:] + 98
	3   Foundation                          0x00007fff4b0d9150 -[NSAssertionHandler handleFailureInMethod:object:file:lineNumber:description:] + 193
	4   AppKit                              0x00007fff465d6f50 -[_NSCGSWindow setFrame:] + 475
	5   AppKit                              0x00007fff4668eb07 _NSCreateWindowWithOpaqueShape2 + 248
	6   AppKit                              0x00007fff4668d763 -[NSWindow _commonAwake] + 1057
	7   AppKit                              0x00007fff46d9bbe7 -[NSWindow(NSWindow_Carbon) windowRefWithCompositedAttribute:andFrameworkScaledAttribute:] + 139
	8   Tk                                  0x00000001061f9ad5 XMapWindow + 239
	9   Tk                                  0x0000000106166dbf Tk_MapWindow + 89
	10  Tk                                  0x000000010616fcc5 MapFrame + 62
	11  Tcl                                 0x00000001060c05cd TclServiceIdle + 76
	12  Tcl                                 0x00000001060a4a96 Tcl_DoOneEvent + 329
	13  Tk                                  0x0000000106145f7d Tk_UpdateObjCmd + 172
	14  Tcl                                 0x000000010603ed6f TclEvalObjvInternal + 773
	15  Tcl                                 0x000000010603ff42 Tcl_EvalObjv + 66
	16  _tkinter.cpython-37dm-darwin.so     0x000000010601a865 Tkapp_Call + 901
	17  python.exe                          0x0000000104e27d14 _PyMethodDef_RawFastCallKeywords + 1476
	18  python.exe                          0x0000000104e34394 _PyMethodDescr_FastCallKeywords + 388
	19  python.exe                          0x0000000104ff89cf call_function + 1535
	20  python.exe                          0x0000000104ff0f15 _PyEval_EvalFrameDefault + 82021
	21  python.exe                          0x0000000104fdcea7 PyEval_EvalFrameEx + 87
	22  python.exe                          0x0000000104e26a29 function_code_fastcall + 377
	23  python.exe                          0x0000000104e25dbc _PyFunction_FastCallKeywords + 668
	24  python.exe                          0x0000000104ff8b5a call_function + 1930
	25  python.exe                          0x0000000104ff0f15 _PyEval_EvalFrameDefault + 82021
	26  python.exe                          0x0000000104fdcea7 PyEval_EvalFrameEx + 87

This was the case for both the master (e9a044e) and the 3.7 (d7cb203) branch. Is there something extra I need to configure for a working build on macOSX?

On my Macbook with Mohave with installed python.org 3.7.2, compiled against tk 8.6.8, the startup header is
Python 3.7.2 (v3.7.2.9a3ffc0492, Dec 24 ...) [Clang 6.0 .. on Darwin].

Your header is quite different (repository?, tk version?) but the version is the exact same: '3.7.2'. This should mean that it is a compilation of released 3.7.2 and does not have any post-3.7.2 patches, such as the fix here. If it did, the version should be '3.7.2+', as it is for my 3.7 compiled today.

Since the problem resulting from a missing closers was fixed on Windows and Ubuntu by the post 3.7.2 patch, I will presume it is fixed on Mac also until presented with clear evidence otherwise.

Tal, does Grant's problem with compiling on Mac look familiar to you?

PR 11307 was moved to bpo-35610.