Title: IDLE shell shouldn't use TABs
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.5, Python 3.4, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Al.Sweigart, Kamushin.Shen, Zero, cben, irdb, kbk, louielu, rhettinger, roger.serwy, terry.reedy, westley.martinez
Priority: high Keywords: patch

Created on 2010-01-11 12:36 by cben, last changed 2017-06-14 09:17 by louielu.

File name Uploaded Description Edit
issue7676.patch roger.serwy, 2011-12-09 01:28 review
fix_indent.patch Kamushin.Shen, 2014-03-18 01:50 review
Messages (18)
msg97586 - (view) Author: Cherniavsky Beni (cben) * Date: 2010-01-11 12:36
IDLE defaults to indenting with 4 spaces in editor windows, but hard TABs in the Python Shell window.  This is inconsistent with PEP 8; what's worse, it's makes copy-paste code between the shell and editor windows confusing and dangerous!

Recently I gave a 3-day Python course to 6 people.  I spent some time explaining the dangers of TABs with Python's indentation-sensitive syntax and telling them "just use 4 spaces and all will be OK".  A few hours later one of them asked me why IDLE is refusing to run his code.  Turns out he copy-pasted code he tried out in the shell, edited it and tried running it, which resulted in a mix of tabs and spaces (which triggered IDLE's tabnanny check) and a mixture of 4 and 8 indents (which is ugly).

I had to explain IDLE's confusing behavior, introduce them to Untabify, and apologize for the inconvenience.  All this TABs stuff (and Untabify) are things people should eventually learn - but they shouldn't bite them on their first day!  (This is what Ubuntu calls a "papercut".)

The rationale for the current behavior seems to be making indentation clear despite the prompt offsetting the first line by 4:

>>> for i in range(3):
        print i

There are 3 alternative behaviors that would be better:

 (1) Use 8 spaces instead of a TAB.
 (2) Use 4 spaces.
 (3) Use 4 spaces, but add a GUI left margin of 4 to continuation lines.

(3) would be ideal, making copy-paste work cleanly while looking good.  I'm not sure if it can be implemented easily, but I'll look into it.

If it's hard, (2) would still be a big improvement IMHO.  I think correct behaviour is more important than looking good after ">>> " - but others may disagree on this.

If there is doubt, (1) is still strictly an improvement.  It can lead to mixed 8/4 spaces - but at least there won't be invisible problems that the user doesn't understand.

[Configurability concerns: all I said above refers to the *default* behavior of IDLE, which should follow 4-spaces.  The user should be able to configure it to use another width, or TABs.  This option already exists - but it's ignored outright by the shell window.  Solutions (2) and (3) would make the shell window respect it.]
msg113991 - (view) Author: Cherniavsky Beni (cben) * Date: 2010-08-15 18:44
This is almost a duplicate of
(though the solution there took a different direction).
msg149068 - (view) Author: Roger Serwy (roger.serwy) * (Python committer) Date: 2011-12-09 01:28
Here's a simple patch to fix this bug.

The ">>> " prompt causes the first level of indented code to use 8 spaces. Further indented code should use 4 spaces, but still uses 8 spaces likely due to the bug described in #8285.
msg151418 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-17 00:55
I closed #8285 as it merely reported a consequence of using tabs.

The Command Prompt window used for interactive Python on Windows only uses fixed-pitch fonts. Idle (tk) allows any font. I happen to use Lucida Sans Unicode (variable pitch) for what I think is the largest set of glyphs. (Perhaps Lucida Console (fixed) has the same set, I don't know. See #13802 for an idea to improve font selection in this regard.) Whatever we do should work well with either fixed or variable pitches. (The current patch does not -- see below.) Or we should prohibit variable pitch fonts and only allow fixed pitch fonts

Selection in CP selects a rectangle, so one can select code with or without the prompts. This is only possible with fixed pitch. Tk text boxes are limited to selecting a continuous stream of text, without excluding a margin.

This contrasts with, for example Notepad++. Its (tabbed) edit window has a gray left margin with line numbers. Then it has a narrow lighter gray column with [-] and [+] boxes to collapse and expand indented regions. Then it has a white text box for user entry and only user entry. Selections ignore the gray margins and only select within the white user text box. Perhaps this is done by tying three borderless windows to one vertical scroll bar, all inside one frame. I don't know. If we could do the same with tkinter, we could put prompts in a grayed margin, but I do not know if we can.

CP has tab stops at every 8 chars. So if one indents with tabs, the first indent is 4 spaces past the prompt and further indents are 8 spaces beyond the last. Not good. The alternative is hand spacing. Also not good if you do very much. Idle (Tk) appears to have tabs at fixed positions, regardless of the font. (Notepad++ acts like IDLE's edit windows -- a tab in converted to 4 spaces.) In Lucida Sans, a tab is physically the same as 16 spaces. Since each '>' is 3 spaces, an indent of 8 spaces instead of a tab is a dedent on the screen. So the patch does not work for me.

My conclusion is that having the shell write a prompt onto the user entry line, in the tk environment, is incompatible with the behavior we want. Fix 1 would be to put the prompt on a line by itself ('>>>\n' or 'input>\n'), use the same indents as the editor, and add an similar output marker ('###\n' or '<output\n'). Option 2 is to omit prompts, use editor indents, and prefix every output line with '#'. Fix 3 would be to put the prompt in a grayed margin. (If this can be done, we could also add line #s as an editor option.)

A separate idea is to give output a colored background.

msg213934 - (view) Author: Kamushin Shen (Kamushin.Shen) Date: 2014-03-18 01:50
I'm a GSOC student. And I submit this patch as a part of my proposal.
This patch solve the copy-paste problem by judging the first line. However, it seems a little ugly in IDLE.
msg215281 - (view) Author: Westley Martínez (westley.martinez) * Date: 2014-03-31 22:46
I think the prompt should be in margins, completely separate from the input.  It is just meant to be a guide anyway, and make this would make it easy to copy code from the interactive shell to a file.  I think output should be in a separate window or frame altogether.  I see no reason for keeping it tied with the input.
msg215288 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-04-01 01:04
> IDLE defaults to indenting with 4 spaces in editor windows, 
> but hard TABs in the Python Shell window.  This is inconsistent 
> with PEP 8; what's worse, it's makes copy-paste code between 
> the shell and editor windows confusing and dangerous!

I also teach Python and find this to be a major PITA.

A Cntl-T turns tabs off but leaves the tab spacing at eight.
msg215289 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-04-01 01:18
Continuing my last post ...
Fix 4 is to put prompts in a narrow window to the left of and kept in synch with the main shell input/output window. This is the approach used by Heblikar in #17535.
msg221154 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2014-06-21 04:46
[Cherniavsky Beni]
> it's makes copy-paste code between the shell and editor 
> windows confusing

This has been a continual source of frustration for students in my Python courses as well.  I'm looking forward to it being fixed.
msg223452 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-19 00:24
Raymond, I cannot program 'fix it'. I need an exact spec. I suspect that any change will upset someone. Which of the four alternative proposals would you consider to be an improvement? One of the problems is that what might look good on screen with colors, including shaded background, might not work as well when cut. We may have to try to separate the two.
msg223467 - (view) Author: Stephen Paul Chappell (Zero) Date: 2014-07-19 15:43
If you want the IDLE shell to be as consistent as possible with the editor windows, changing the TAB binding to insert four spaces instead of a tab (alternative 2) would be helpful.
msg223485 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-19 22:49
The problem with just /\t/    / is the absence of a secondary prompt.

>>> def f():
    return 'not acceptible'

Hence the 8+4+4... proposal. With a proportional font, the return would start to the *left* of def. Adding '... ' works for fixed pitch fonts, but '...' is much narrower than '>>>' in a proportional font.

The problem with mandating 'fixed pitch' is that unicode fonts, such as Lucida Sans Unicode on Windows, are proportional.  With that font, spaces are very narrow: '8+4' would need to be 13 + 4 + 4 ... to look right, *on Idle*.  I feel that continuing to work with proportional fonts, for code not limited to ascii, is more important than this issue.

To look right regardless of font, code should start at the margin, with prompts in a separate narrow window. This means that output would not be dedented under input prompts.
 in> def f():
         print('code against margin')

out> code against margin
' in> ' and 'out> ' (or whatever input and output indicators were used) would be in a vertical strip (like line numbers) with, say, a light gray background (such as used here).  Normal output would be, say, the current blue on a very light blue background. Similarly for red stderr output. It takes very little tint to be distinct from a white background. I have set my custom configuration to try this out.

Cutting code is a different issue. As I said above, I think custom functions are needed. One could only cut code. Another could add '>>> ' and '... ' before input lines, for pasting into docstrings. Another could add '# ' before output lines, for pasting into messages in a form that readers could recut and paste into an editor.
msg223541 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-07-20 21:51
To continue and expand some things I said in various messages above...

The reason Shell does not and indeed should not have a secondary line prompt is that in Shell, '>>> ' means 'Enter a statement:' rather than just a line. Being able to enter, edit, and recall complete statements rather than just lines is an important feature of Idle's Shell.

For proportional fonts, I suspect the tab stops are every 8 em quads. A em quad (named after M) is square, so it is as wide as the font is high.

In msg151418 I suggested that the solution is to not mix prompts with user input, as is done. I gave the narrow prompt window as fix 3. I suggested as fix 1 to put statement prompts and output indications on separate lines. Here is a mockup example.

def f(x):
    if x:
        print('got it')
        return 'something'
got it

(The 'blank line' signaling 'end of statement' is intentionally suppressed.) This should be easy to do and, to me, looks pretty decent. The hesitation I had before is the copy for paste problem, but if that is a custom function, it would not be hard to reformat. A minor problem is the output could include '>>>:\n' or '---\n', but today one can do

>>> print(">>> def fake():\n\treturn 'def'\n")
>>> def fake():
	return 'def'

msg228719 - (view) Author: Stephen Paul Chappell (Zero) Date: 2014-10-06 16:24
In Lib\idlelib\, there are usetabs and indentwidth attributes in the PyShell class. Is there some reason that these settings cannot be reconfigured in the Options > Configure IDLE... menu? I just edited these to False and 4 respectively and am quite content with the resulting behavior.
msg228730 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-10-06 19:50
They potentially could be.  However, if the result is something like
>>> if a:
    if b:
or worse, with Lucida Sans Unicode, the visual equivalent of
>>> if a:
  if b:
then I do not find result very satisfactory.  I think moving the prompt to a sidebar, as with line numbers (forthcoming), is a better solution.  This will make code look good both in the shell and when copied and pasted elsewhere.  I hope to have a patch in a month or two.
msg233701 - (view) Author: Al Sweigart (Al.Sweigart) * Date: 2015-01-09 00:43
There are three pieces of user-specified configuration: (1) the number of spaces for a tab set in IDLE's config, (2) the sys.ps1 value, and (3) the sys.ps2 value.

Currently IDLE's shell is ignoring (1) while the editor is not. IDLE's shell is ignoring (3) while the python shell isn't. I think the obvious solution is to make IDLE's shell follow the same behavior as IDLE's editor and the python shell.

The draw back comes from variable-pitch fonts, but I think using variable-pitch fonts is a minority use case and will create spacing issues regardless (compared pasting 8 spaces and pasting 1 tab into the editor with Lucida Sans Unicode). Special cases aren't special enough to break the rules.

This issue is concerned with mixed tabs and spaces in the IDLE shell, which has a simple fix. Putting the >>> and ... prompts in a separate space so that they are not copied can be done in a separate issue.
msg233705 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-01-09 01:02
I believe I explained above the logical and technical factors that make the Idle shell inherently different from the console shell in this regard.  As the title says, this issue is about not using tabs and not about mixing tabs and spaces.  As I already said, my intended fix is to put the prompt in a sidebar (adapting the proposed editor line number code), which will allow use of space indents in the entry area, as in the editor.
msg295996 - (view) Author: Louie Lu (louielu) * Date: 2017-06-14 09:17
In #30663, I add the line number sidebar to editor window. I've attempted to do the same in pyshell, something different that need to concern is the indent prompt showing.
Date User Action Args
2017-06-14 09:17:55louielusetnosy: + louielu
messages: + msg295996
2015-11-20 06:21:49irdbsetnosy: + irdb
2015-01-09 01:02:31terry.reedysetmessages: + msg233705
2015-01-09 00:43:58Al.Sweigartsetmessages: + msg233701
2015-01-08 12:15:06Al.Sweigartsetnosy: + Al.Sweigart
2014-10-06 19:50:32terry.reedysetmessages: + msg228730
2014-10-06 16:24:18Zerosetmessages: + msg228719
2014-07-20 21:51:20terry.reedysetmessages: + msg223541
2014-07-19 22:49:31terry.reedysetmessages: + msg223485
2014-07-19 15:43:48Zerosetmessages: + msg223467
2014-07-19 00:24:49terry.reedysetassignee: kbk -> terry.reedy
messages: + msg223452
2014-06-21 04:46:54rhettingersetpriority: normal -> high

messages: + msg221154
2014-06-05 16:50:49terry.reedysetversions: - Python 3.2, Python 3.3
2014-06-05 16:32:33Zerosetnosy: + Zero

versions: + Python 3.4, Python 3.5
2014-04-01 01:18:06terry.reedysetmessages: + msg215289
2014-04-01 01:04:04rhettingersetnosy: + rhettinger
messages: + msg215288
2014-03-31 22:46:11westley.martinezsetnosy: + westley.martinez
messages: + msg215281
2014-03-18 01:50:44Kamushin.Shensetfiles: + fix_indent.patch
nosy: + Kamushin.Shen
messages: + msg213934

2014-02-04 12:05:41taleinatsetnosy: - taleinat
2012-01-17 00:55:03terry.reedysetversions: - Python 3.1
nosy: + terry.reedy

messages: + msg151418

stage: needs patch -> patch review
2012-01-16 22:32:41terry.reedylinkissue8285 superseder
2011-12-09 01:28:43roger.serwysetfiles: + issue7676.patch
versions: + Python 3.3
nosy: + roger.serwy

messages: + msg149068

keywords: + patch
2010-08-15 18:44:15cbensetmessages: + msg113991
2010-07-20 15:33:45taleinatsetnosy: + taleinat
2010-07-11 11:47:50BreamoreBoysetassignee: kbk
stage: needs patch

nosy: + kbk
versions: + Python 3.1, Python 2.7, Python 3.2
2010-01-11 12:36:19cbencreate