This issue proposes to add a shell-specific sidebar to Shell.  It follows-up #17535, which added a line-number sidebar to module editors and implements the first phase of #37892, which proposes that the single statement editing area of Shell should be strictly just that.

Looking ahead to this issue, #17535 added both a generic Sidebar class and a specific LineNumber subclass.  This issue proposes to add a 3 char wide ShellIO subclass that would mark the beginning of the blocks of text added to the Shell history area. Labels, such as '>>>', 'Out', 'Inp', and 'Err' would be used for the first line of user code input, program stdout, user response to input() (there is only one line), and program stderr (which includes tracebacks).  I am not quite sure what to do with debug notices and warnings from the warning modules.  Maybe use 'Con', maybe use Dbg and Wrn.  I expect to test variations.

As with LineNumber, the font face and size will the the same as in the text.  The labels should use the highlight colors of their text block except that '>>>' should continue getting the 'console' colors.

I thing the initial implementation should not response to clicks.  After experimentation, we might decide that clicking on a label could select whole blocks, but I would like get the essential stuff right first.

See PR GH-15474 with an implementation.

The documentation for sys.ps1 and sys.ps2 states that they "are only defined if the interpreter is in interactive mode." Since the IDLE shell is meant to be interactive (and to reduce the differences between the shell and running Python directly), would it be helpful if ps1 and ps2 were defined when running IDLE? The shell could then honor their values.

If such a direction was explored, one issue may be that the sidebar could not simply be 3 char wide. The documentation also states that non-strings are evaluated each time they are needed by the interpreter. This allows for such interesting possibilities as shown with the code below but may not be desired functionality for the IDLE shell window.

import sys
from datetime import datetime

class TimePrompt:
    def __init__(self, prefix, suffix):
        self.prefix, self.suffix = prefix, suffix
    def __str__(self):
        return f'{self.prefix}{datetime.now()}{self.suffix}'

sys.ps1, sys.ps2 = TimePrompt('', ' >>> '), TimePrompt('', ' ... ')

Will the current interactive shell continue to be available?

Stehpen, I'm not sure what your use-case is... To me it seems that sys.ps1 (and ps2?) aren't quite relevant any more when using a sidebar, since there are no longer inline prompts.

Raymond, with the current PR, this replaces the inline prompts. Making it toggle-able could be very awkward. Is this a deal-breaker in your opinion?

Maybe my impression has been false this whole time, but the Python interactive interpreter seems to be very similar to the IDLE shell window. My question is, "Why not make them even more so?" Having IDLE react to sys.ps1 and sys.ps2 opens up the shell to similar use cases as having them defined in the interactive interpreter. My suggestion is not to have them added as text as is usual in a terminal window but to print the values of ps1 and ps2 in the proposed ShellIO subclass. That would make label customization possible in a way that is already documented and standard.

Stephen, perhaps what you're suggesting is out of context for this issue? This issue is about removing prompts from the shell window's text widget, replacing them with a separate sidebar showing where prompts and user input were. This will make IDLE's shell less similar to the basic command line shell, not more similar.

Zero:     "not to have them added as text as is usual in a terminal window"
taleinat: "removing prompts from the shell window's text widget"

Zero:     "print the values of ps1 and ps2 in the proposed ShellIO subclass"
taleinat: "separate sidebar showing where prompts and user input were"

We appear to be in agreement.

terry.reedy: "Labels, such as '>>>', 'Out', 'Inp', and 'Err' would be used"
Zero:        "Having IDLE react to sys.ps1 and sys.ps2"

My suggestion is to take those labels terry.reedy talks about from the values of ps1 and ps2 since they are already documented and standard for "the interpreter ... in interactive mode." If psX needs to introduced for other prompts that may be needed ("Maybe use 'Con', maybe use Dbg and Wrn."), it would provide a sensible way to customize those labels as well.

After the PR, can someone easily  reproduce the interactive prompt sessions in the tutorial and maindocs.  Will it look the same?  Would they be able to build doctest examples in the IDLE shell?

This is marked for 3.7 through 3.9.  Do you intend to backport this PR?

> After the PR, can someone easily  reproduce the interactive prompt sessions in the tutorial and maindocs.

Yes, except with prompts (both >>> and ...) in the sidebar rather than inline.

> Will it look the same?

No, see above.

> Would they be able to build doctest examples in the IDLE shell?

That's a good thought. It should be rather simple to add a context menu option to "copy for doctest" which would add the prompts to the copied text.

(By the way, how popular are doctests these days? My impression is that they've completely fallen out of style, but I'm sure that could vary between different areas and industries.)

Stephen (Zero), thanks for the clarification.

FWIW I'm -1 on the ability to customize the prompts in the sidebar. It would complicate the implementation while reducing the straightforwardness and simplicity of IDLE.

Also, anyone who *really* wants to customize them could just change the hard-coded values in the code in their own copy of IDLE. I don't think it's worth the effort to make it more easily configurable.

> This is marked for 3.7 through 3.9.  Do you intend to backport this PR?

IDLE generally has a different backporting policy: We generally backport all IDLE development to all versions of Python that aren't in security-only or unmaintained status, except for 2.7. Currently, that means 3.7 and above.

There is always room for exceptions, of course.

> By the way, how popular are doctests these days?

Arguably, Sphinx has made them more popular than ever.  For me, they are essential.  And as long as doctest remains a non-deprecated module, we have to retain support.

For the record, I'm opposed to backporting this to 3.8 and 3.7.  If this PR ends-up killing our teaching workflow, we need something to fallback to.  Also, you don't want to immediately invalidate every video or written tutorial that uses IDLE for demonstrations.

> Raymond, with the current PR, this replaces the inline prompts. 
> Making it toggle-able could be very awkward. Is this a 
> deal-breaker in your opinion?

I can't tell for sure because I'm having a hard time building with Tkinter support, so I can't easily try out the PR.

As stated in my second opening sentence, this issue "implements the first phase of #37892", which gives the context and specific rationale.

There are 3 separate parts to a Python Shell.

1. Code entry, indentation, display, and copying.  This issue fixes Shell design flaws that have been recognized as such for at least 15 years.  In the process, it makes Shell *more similar* to the standard REPL in important respects.  In particular, by default, compound statements will look the same in Shell as the same code entered into REPL with 4-space indents.  It will also be possible to cut and paste into an editor have it look right.

Raymond: I do not plan to retain a buggy shell mode that in #7676 you called a "major PITA" (msg215288) and "a continual source of frustration for students in my Python courses" that you were "looking forward to it being fixed."

2. Code execution.  IDLE executes syntactically correct user code so that, to the extent possible and sensible, the result is the same as if executed directly with python.  Not an issue here.

3. Everything else.  Alternate shells are intentionally difference from and hopefully improve upon the interactive python.exe REPL.

Paul: The existence of two prompts, sys.ps1 and sys.ps2 in python.exe interactive mode is a kludge needed to interface nested multiline statement Python with *single text line entry* system terminals.  I believe that most beginners never fiddle with them.

IDLE's Shell is a GUI-based python-statement-aware interface.  The python.exe executing user code from shell or editor is never in interactive mode.  For statements entered and executed, '>>>' is quite sufficient to mark code.  The sidebar should be minimal and unobtrusive and 3 columns is sufficient.  So I reject the idea of expanding it.

#37892 mentions the possibility of later adding an expanded full-line prompt for the code entry area, such as "Enter below a complete Python statement."  When we get to that, we can consider an option (for non-beginners) to customize it.

To help get a feel for what the current PR looks like, I'm attaching a screenshot (taken on Windows 10).

Thanks.  Can you also show a script running with F5 showing a restart, the output from the script and a subsequent interactive session that shows the inspecting variables.  Also show what an exception and SyntaxError looks like.

My previous response was to comments up to msg355812.

To answer Raymond's subsequent questions: The user will enter python code the same as before.  Currently, the indents and resulting appearance of the code on Shell screen is different from (and worse, sometimes far worse, than) the appearance of the same code in the REPL.  After this patch *and* a required immediate follow-up to replace tab indents with, by default, 4-space indents, the code on the screen *and in saved sessions* will appear the same as in the REPL (when 4-space indents are used).

In Tal's screenshot, the appearance is better, but the indents are still tabs.  Because of the regression being fixed by #38636, PR 17008, he could not manually switch with alt-T and alt-U.

(The current sidebar patch is also missing output markers.  Also, for my custom dark theme, I colored the sidebar dark blue on light bluish gray and the result, to me, is aesthetically much more pleasing than in the screenshot.)

The improvement is, to me, even more evident in the following pair of examples -- first REPL and improved IDLE, second current IDLE. 

>>> if True:
...     print('true')
... else:
...     print('false')

versus

>>> if True:
        print('true')
else:
        print('false')

#7676 was marked as a bugfix issue and I regard changing examples like the above to be a long-overdue and badly needed fix.  Any version that does not get this fix will also not get followup changes.  I just added msg355853 to #37892 to list some I have thought of.

One of my intended followups is to add multiple ways of saving a shell session.  IDLE can juggle prompts/markers, code, and output to match 'templates' much easier than users can.  I had not thought about doctests, but saving shell interaction in the proper format should be an easy addition.

SyntaxErrors aside, output will be the same as it is in IDLE now, with 'Out', 'Err', and maybe 'Inp' markers ('Imp' for user code input(prompt) lines) planned for the sidebar.

IDLE already handles syntax errors in the code being entered differently from the REPL by replacing the caret line with an error highlight, as in the editor.  I would like it follow the logic of not imitating dumb terminals by putting the cursor at the marked spot, as in the editor, and letting the user directly correct the problem.  Messing up the input with a usually useless traceback and a new prompt and requiring retrieval of the statement with the marker removed and the cursor in the wrong place is a nuisance.  I also don't see any reason to burden the session history with bad code that was never submitted for execution.  (Note that IDLE does not RESTART when code in the editor has syntax errors.)

Strong +1. I should have done this 20 years ago.

With the support from Guido and others here and on idle-dev, Terry and I will be moving forward with this change.

However, I take Raymond's concern about this "killing our teaching workflow" seriously. Raymond, let's please work to figure out how to make sure this improves, rather than degrades, the teaching of Python with IDLE. I'd be happy to work on features, updating documentation, outreach, or anything else that this would require.

Do you still plan to backport to 3.7 and 3.8?

Yes.

This feature may be fine or it may not.  I haven't been able to apply the PR and get a working Python 3.9 with Tkinter support, nor have I seen the requested additional screen shots.

Ideally, major changes in appearance or functionality should have significant user testing and feedback.  However, the tone of this thread is that the user feedback is irrelevant and the feature will happen and be backported regardless.

My team has used IDLE 8 hours a day for the last eight years, cumulatively training 25,000+ engineers using IDLE.  We have served as the beta tester and as the canary in the coal mine, reporting every usability issue so that IDLE can be improved.  We have tried to be a voice for users and it is sad to be roundly dismissed.

Other bug reports and feature requests that matter to us don't appear to be getting any traction:

* The squeezer on-by-default degraded our user experience
* Need auto-save shell sessions (we record full day sessions)
* Print speed is slow, giving the appearance of Python being slow
* Need Alt +/- hot keys for changing font size
* Recently lost the strip-trailing-whitespace menu option
* Need auto-strip-trailing-whitespace on save
* Tab completion has been unreliable for several years
* Need an option to run the install-certificates script (otherwise urllib HTTP requests fail with a fresh install of Python from python.org)
* Cntl-A on macOS jumps before the >>> prompt
* Font samples on the Settings tab are useless, distracting, and so large that it can make it impossible to mouse down to the Ok or Apply buttons when the display size is small (constrained by the projector).
* The tab size slider should be a scroll box and the font size scroll-box should be slider (users should almost never change the tab size but almost always want to change the font size).
* The default font size is too small.  For some users, when IDLE opens, the text looks like dust.
* The completions pop-up menu has never been helpful and users have a hard time getting it dismissed (so we usually have people change the completions pop-up time to 10,000 milliseconds)
* The mouse cursor targeting frequently gets missed by about seven lines (my understanding is that this is an upstream Tkinter problem, but it has been around for many years -- we have to close and reopen windows frequently to make this goa away).
* For the past two or three years, we periodically get an unintentional clipboard paste in to the shell window.  This happens during live demos, so the instructors have been accustomed to reflexively type Cntl-Z to undo the paste so as not to interrupt the demo).
* For mac users who have enabled tabs system-wide, opening a new window in IDLE triggers a system log-out (amazing that this is even possible).  My understanding is there isn't anything we can do about this, but it is a really bad out-of-the-box user experience seconds after a fresh install of Python.
* On rare occasion, I teach Python to kids.  A menu option to launch the turtle demo would be helpful.
* Being able to run pip from within IDLE would be a major win, especially for Windows users who are foreign to the command-line and who often do have Python on the PATH.
* The Search Dialog window doesn't stay on top of other windows.  It can have the focus, but be hidden underneath other windows.  The user experience that is there is no key that they can press to continue running their session.
* Sometimes the Search Dialog window becomes unresponsive and there is no way to clear it without turning IDLE off.
* When launching IDLE from the command-line, we see various exceptions and tracebacks for errors in the code (typically along the lines of None-type object doesn't have some attribute).
* Several of these issues have only came-up in the past few years.  IDLE used to be more stable than it is now.

@rhettinger: The turtle demo is easily accessible through the menus via Help > Turtle Demo.

It is nice to see there are others interested in IDLE's improvement. :-)

> The turtle demo is easily accessible through the 
> menus via Help > Turtle 

Thanks.  I didn't see that one land.  It is a nice improvement.

Hmm, the turtle demo is barely usable on the mac:

* The "Choose Example from Menu" is a label and not clickable.
* The left pane is clickable but does nothing.
* When code is chosen from the menu the font-size is small.
* The "start" button label is invisible and the "stop" and "clear" buttons are barely visible with yellow-on-white.
* Once started, the "stop" button is invisible and the other two buttons shift to yellow-on-white.

@raymond Please stop spamming this issue with unrelated IDLE issues.

> Can you also show a script running with F5 showing a restart, the output from the script and a subsequent interactive session that shows the inspecting variables.  Also show what an exception and SyntaxError looks like.

I've attached a screenshot showing everything that Raymond asked for.

Raymond, FWIW, this will fix at least one of the issues you've mentioned: "Cntl-A on macOS jumps before the >>> prompt".

The base issue #37892 is about moving the IDLE Shell prompt away from the first line of user code input, so that code starts at the margin where continuation lines start and we can use space indents.  I said the prompt could either go left, into a sidebar, or up, on a line by itself.

I opened this sub-isssue for the first option and Tal surprised me a few days later with a patch.  I expected a few months of development, but we were stuck last fall with mysterious Pipelines failures on Ubuntu and Windows.  I discoverer yesterday that the failures were due to Pipelines stealing sys.stdout output that was supposed to go to the Shell being tested.  The tests now pass and I believe PR 22682 is about ready for the beta next Monday.

Today I implemented the second option; it only took 3 simple changes on 3 lines.  PR 25678 for #37892.  I have tested with single and multiline statements, without and with debugger on, and all seems ok so far.  I am not surprised because indents now use the well-tested editor branch in the newline_and_indent_event method. So I think this is also ready for the beta.

For me, either fix is a pleasure to use and a relief, like having tiny thorn removed after 15 years. Anyone with a local repository can make a couple of branch to test now.  Or wait until next week.

My intention is to merge one or both options into 3.10.0b1 and allow user testing and feedback, further patching, and another beta release before backporting to 3.9 (perhaps late next June).  It is too late for 3.8.

Tal, I completely agree with not switching between straight margin and space indents versus the mis-design of an uneven margin and tab intents.  But I want the option of using either improved mode for myself as well as others.  I think Shell will have be cleared and restarted when switching.

Raymond, about interactive sessions for the docs: some have secondary prompts, some do not.  The sorting how-to uses the former.  (Would you like some .rst markup included?)  I don't know if the latter always have the same format.

I have long thought that there should be several output formats for a shell session, including one ready to run.  But no patches yet that I know of, or even any exactly specified requests.  Anyway, anything possible before should remain possible.  What is needed is a shell parser (for each mode) that would yield a sequence of code, traceback, and output objects.  Then various document builders could be written.

New changeset 15d386185659683fc044ccaa300aa8cd7d49cc1a by Tal Einat in branch 'master':
bpo-37903: IDLE: Shell sidebar with prompts (GH-22682)
https://github.com/python/cpython/commit/15d386185659683fc044ccaa300aa8cd7d49cc1a

To those following, please note that the latest PR (GH-25708) adds a "Copy with prompts" context-menu option, to make copying interactive sessions for doc-tests and code examples convenient.

There is also a 'copy only code' option.  Like 'copy', these two new options require a selection.  They appear on the context menu when right-clicking either on the sidebar or within the selection.  Like the format region commands, these new options operate on a 'region', a sequence of complete lines in the text area.  And as with region reformatting, incomplete regions are expanded.  Regions are easily selected by left clicking and dragging on the sidebar, as with the line number sidebar.  

Two of us reviewers found 3 bugs; 2 have been fixed and the 3rd should be soon.  I imaging that we have found everything, but more testing today, after the 3rd fix, by others, would be good.

The "copy only code" feature has been removed for now.  It can't currently be properly implemented, due to code input not being marked different than user input in between REPL commands, e.g. in response to an input() call.

With that, the PR is currently free of known bugs. I'd be very grateful to anyone who would try it out and give some feedback!

Also, presumably most users don’t right click and look for options. They
just drag a selection and hit ^C (or cmd-C).--
--Guido (mobile)

New changeset b43cc31a270d0dacbc69e35d6c6fbdb5edd7e711 by Tal Einat in branch 'master':
bpo-37903: IDLE: add shell sidebar mouse interactions (GH-25708)
https://github.com/python/cpython/commit/b43cc31a270d0dacbc69e35d6c6fbdb5edd7e711

Is it possible to add a Settings option under the General tab to switch between the sidebar mode and traditional mode?

As described above in msg392163, the other fix is to put the prompt on a line by itself.  I hoped to get a switch between fixes into b1 but ran out of time and energy.  I intend to have in in the next, along with other improvements.

> Is it possible to add a Settings option under the General tab to switch between the sidebar mode and traditional mode?

Raymond, that should be possible, and possibly rather easily. I'll take a look at adding this option.

Upon reading Terry's previous comments here more carefully, it's clear that he will be adding an option to switch between this mode and an alternate mode with prompts on separate lines in the main text widget. The latter will be much more similar to the traditional mode.

Rather than request a specific solution, I'll state what new problems need to be solved.

For teaching purposes in live demos, it is essential to have a clear visual distinction between the inputs and outputs:

    >>> beatles = ['john', 'paul', 'ringo', 'george']
    >>> [name.capitalize() for name in beatles]
    ['John', 'Paul', 'Ringo', 'George']
    >>> [name for name in beatles if 'n' in name]
    ['john', 'ringo']

This doesn't work nearly as well:

    beatles = ['john', 'paul', 'ringo', 'george']
    [name.capitalize() for name in beatles]
    ['John', 'Paul', 'Ringo', 'George']
    [name for name in beatles if 'n' in name]
    ['john', 'ringo']

With the sidebar, the ps1 prompt helps a little, but gray separator bar visually runs the input and output text together and even causes consecutive inputs to visually be merged.  The strongest separator, the unindent, has been lost.

    >>>|beatles = ['john', 'paul', 'ringo', 'george']
       |[name.capitalize() for name in beatles]
    >>>|['John', 'Paul', 'Ringo', 'George']
       |[name for name in beatles if 'n' in name]
    >>>|['john', 'ringo']

I believe that if you consult a trained graphic designer (i.e. one who can name the 7 elements of graphic design and describe how they are used), they will confirm that the new display is problematic.  To make data groups (inputs and outputs) visually distinct, we can change alignment, change vertical spacing, remove strong vertical lines, change color, etc.).  But if you align the text and run a vertical line separating it from the PS1 prompt, then the distinction is blurred.  At least this is what I was taught in graphic design courses.  

Almost the entire purpose of graphic design in an IDE is to help the eye either group together like pieces of information or to help the eye create distinctions between dissimilar things.  We give elements the same color if they are related and different colors if unrelated.  We bring like information together by placing it close together or by giving it the same alignment.  We push things apart and disconnect them by drawing a separator line (in the case at hand, the ps1 prompts are clipped-off in a separate window from the text they were intended to annotate).

If we have to keep the sidebar, some vertical separation would help (as it does in ipython):

    >>>|beatles = ['john', 'paul', 'ringo', 'george']
       |[name.capitalize() for name in beatles]
       | 
    >>>|['John', 'Paul', 'Ringo', 'George']
       |[name for name in beatles if 'n' in name]
       |
    >>>|['john', 'ringo']
       |

This added spacing would help separate consecutive statements but would cost eating up valuable vertical space and would not help with visually distinguishing the input from the output.

You could take out the vertical line and get an improvement:

    >>> beatles = ['john', 'paul', 'ringo', 'george']
        [name.capitalize() for name in beatles]
       
    >>> ['John', 'Paul', 'Ringo', 'George']
        [name for name in beatles if 'n' in name]
       
    >>> ['john', 'ringo']

This is better, but could be improved by unindenting to distinguish the inputs and outputs, which just about takes us back to where we started from (not just in IDLE, but what you see in books, presentations, and blog posts):
       
    >>> beatles = ['john', 'paul', 'ringo', 'george']

    >>> [name.capitalize() for name in beatles]
    ['John', 'Paul', 'Ringo', 'George']

    >>> [name for name in beatles if 'n' in name]
    ['john', 'ringo']

Another desirable feature is the ability to cut and paste snippets into docstrings.  This is essential not just for doctest, but even for untested examples in docstrings.
Those examples, need to be recognizable to humans as interactive sessions with outputs clearly distinguished from inputs.  When I tried out the new sidebar in prep for a class, it was impossible to include the ps1 prompts in a cut and paste selection.  They had to be manually typed back in.  

Lastly, for live demos in presentations, it is desirable to have a clear screen, free of distracting artifacts.  The sidebar is such an artifact -- you would never see such a thing in a book or slide presentation.  Also, for live demos, large fonts are necessary which means that screen space is at a premium and the loss of horizontal space matters.

Yes, lets discuss actual problems and various possible solutions.

Raymond's example is incomplete as it omits multiline statements, which is where the real problems lie and the reason a fix was sorely needed.  This has been more or less agreed on since at least 2005.

Old:
------------------------------
>>> if a:
        print('okay:'
else:
        if a is None:
                print('none:')
        else:
                print('else:')

else:
------------------------------

Possible confusion between input and output is a pre-existing issue.  For the default themes, Shell User Output has the same background color as Code or Normal Text.  The foreground is blue.  The lack of contrast is worse on the dark theme.  I think using the same background was a mistake, but so far changing the default has been considered off limits. The pre-existing fix for distinguishing output is to customize colors for Shell User Output, as I have done. I have also given Shell User Exceptions more contrast.  One can make both as different as one wants.

In 2014, #7676, Raymond said that the indent mess was a "major PITA" and the impossibility of copy-pasting correctly indented code into an editor  "has been a continual source of frustration for students in my Python courses as well.  I'm looking forward to it being fixed."

At the time, I had no more idea how to fix this than Kurt Kaiser did in the 2000s.  But I eventually realized that the problem is entirely due to the first line of multiline statements being indented relative to following code lines, unlike the REPL.  The fix (and the only acceptible fix) was and is to stop doing that.

The day before the beta1 release*, Tal and I merged a beta-quality release that we said would need improvement before production release.
------------------------------
>>> if a:
...     print('okay:'
... else:
...     if a is None:
...         print('none:')
...     else:
...         print('else:')
...
    else:
------------------------------
I am sure that a graphics designer would consider this improved.  Unindented code now lines up, and the indents are that same as in any decent python-aware editor.  In any case, the above is identical to the REPL except that in the REPL, the output starts at the left, under the prompts.

If one selects the above lines, easily done by clicking and dragging on the sidebar, just as with line numbers, right clicks, and selects 'Copy with prompts' on the context menu, one gets the above *with*  output 'else' to the left.  IDLE is now as useful for interactive snippets as the REPL.  This is mentioned in the IDLE What's New and News items.

I can imagine a possible alternative fix that would put output under the prompts in Shell, not just in the clipboard copy.  It would involve a fixed input prompt above the code entry area and side prompts in the history area.  This would be a variation on the two-text solution that I discussed on #37892.

As for copying code to paste into an editor: As I believe was mentioned above, we had 'Copy only code' on the context menus, but removed it because it also copied user input in response to user code prompts, which is currently tagged the same as code entered in response to the Shell's prompts.  Fixing this will be a high priority issue.  When it is, we will list both new option on the Shell menu.

I will soon open another issue for the sidebar colors.  I consider the current colors to be placeholders.  We set that aside to get the sidebar otherwise ready.  Some might prefer a background matching the general shell background so that there is no vertical separation.


* Not by choice.  We were delayed 6 months because Azure Pipelines, on Ubuntu and Windows, changes the test environment by grabbing sys.stdout.  Shell receives input on sys.stdout.  So when test code sent text to shell via sys.stdout, it went to Pipelines instead of Shell, breaking the tests.  We should have ignored those failures, but didn't.  Once we realized the problem, less that 2 weeks before the beta, we had the test steal sys.stdout back for each test and moved forward.

> For teaching purposes in live demos, it is essential to have a clear visual distinction between the inputs and outputs:
>
> >>> beatles = ['john', 'paul', 'ringo', 'george']
> >>> [name.capitalize() for name in beatles]
> ['John', 'Paul', 'Ringo', 'George']
> >>> [name for name in beatles if 'n' in name]
> ['john', 'ringo']

Thanks for bringing this up. Lack of clear separation of inputs and outputs could be considered a drawback of putting prompts in a sidebar. 

Considering that the prompts *are not part of the code*, I do believe separating them more clearly is a benefit.  To support this, consider that in Jupyter notebooks, the prompts are also placed in a graphically-separated area to the left of both input and output "cells".

Also, do note that IDLE colors outputs differently from inputs, which helps differentiate them from inputs very clearly IMO.  Therefore, the examples in messages posted here are lacking in this regard.  For context, I'm attaching an image with screenshots before and after the change to use a sidebar.

Also, consider that previously, separation of outputs from inputs was not great for multi-line statements, as indentation was inconsistent and more vertical space was used:

>>> if True:
	print('Foo')
else:
	print('Bar')

	
Foo


I think that moving prompts into a sidebar is an improvement in this regard:

>>>| if True:
...| 	print('Foo')
...| else:
...| 	print('Bar')
...| 
   | Foo


Taking these three points together, I the sidebar clearly improves the separation of prompts from actual code, and the differentiation of outputs from inputs does not seem much worse, and is arguably overall better.

For a wider context, besides the default REPL, I also consider two other leading REPLs: ipython and Jupyter notebooks.  They both also have clear separation of outputs from inputs, though in different ways than the standard REPL.


ipython retains similar behavior to the standard REPL in this regard, replacing the '>>>' prompts with "In [1]:", "In [2]:" etc.  It also prefixes outputs of statements with e.g. "Out[2]:", but not outputs written to stdout.  And it has additional vertical spacing between commands.

Here is a short example session:

Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.18.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: if True:
   ...:     print('Foo')
   ...: 
Foo

In [2]: beatles = ['john', 'paul', 'ringo', 'george']

In [3]: [name.capitalize() for name in beatles]
Out[3]: ['John', 'Paul', 'Ringo', 'George']


Jupyter notebooks are a different beast since their interface is browser-based rather than terminal-based, and they take more advantage of non-text graphical elements than IDLE's shell.  Inputs are done in "cells", which may contain multiple statements, and each cell its own output.  These cells are clearly visually separate from each other.  I've attached a screenshot of a similar Python 3 session in a Jupyter notebook for comparison.


Indeed, both the ipython REPL and Jupyter notebooks go further than the default REPL in differentiating outputs from inputs, using different methods.

> Another desirable feature is the ability to cut and paste snippets into docstrings.  This is essential not just for doctest, but even for untested examples in docstrings.
> Those examples, need to be recognizable to humans as interactive sessions with outputs clearly distinguished from inputs.  When I tried out the new sidebar in prep for a class, it was impossible to include the ps1 prompts in a cut and paste selection.  They had to be manually typed back in.

This is no longer the case.  In a quick followup PR, which has been merged for some time now, we added a context-menu option "Copy with prompts" which results in code samples directly usable for doctests and code samples.

> Lastly, for live demos in presentations, it is desirable to have a clear screen, free of distracting artifacts.  The sidebar is such an artifact -- you would never see such a thing in a book or slide presentation.  Also, for live demos, large fonts are necessary which means that screen space is at a premium and the loss of horizontal space matters.

I am not sure I agree with the strength with which you state "you would never see such a thing in a book or slide presentation".  With Jupyter notebooks being very prevalent these days, samples very often include the non-code visual artifacts native to that format.  Arguably, the *prompts* are a distracting visual artifact; the ability to cut them out of screenshots from Jupyter notebooks thanks to the clearer separation of cells and their inputs and outputs is indeed a boon, and we could consider whether we can do something similar for IDLE.

As for lost vertical space, that is indeed at a premium for books and presentations.  However, ipython significantly increases the vertical spacing to the left of code inputs and non-stdout outputs compared to the standard Python REPL and IDLE.  And the sidebar adds 3.5 characters' width only for outputs, which can be avoided when it's truly crucial using the "Copy with prompts" feature.  Overall, considering the benefits, this seems like a relatively minor drawback.

> Other bug reports and feature requests that matter to us don't appear to be getting any traction:

I'll start going through those, and see if I can fix some of the bugs on that list.

A few notes:

> Need an option to run the install-certificates script

Are you suggesting that we add a way to do this directly from IDLE?

> Sometimes the Search Dialog window becomes unresponsive and there is no way to clear it without turning IDLE off.

Is there an open issue about this? Details would be helpful.

> Several of these issues have only came-up in the past few years.  IDLE used to be more stable than it is now.

This may be true, and at least partially due to a continuous need to fix things broken by various OS and Tcl/Tk changes.

> For mac users who have enabled tabs system-wide, opening a new window in IDLE triggers a system log-out (amazing that this is even possible).  My understanding is there isn't anything we can do about this, but it is a really bad out-of-the-box user experience seconds after a fresh install of Python.

This is a prime example of the above: macOS changes + Tcl/Tk issues with those mean we are often very limited in our ability to properly address such issues.  FWIW we added a warning in the IDLE shell about running with that OS option enabled, see bpo-34864 and PR GH-10464.  I wasn't aware of such crashes; is there an issue about them?

> Being able to run pip from within IDLE would be a major win, especially for Windows users who are foreign to the command-line and who often do have Python on the PATH.

I agree!  This needs to be properly designed and would be far from trivial to implement and test.  If an issue about this exists or would be created, I'd be happy to help move it forward.

I will link to an IDLE shell followup issue later.

Discussions about other IDLE issues should be on pre-existing issues (use the tracker search with Components: IDLE), new issues, or without an issue, on the idle-dev list.