classification
Title: IDLE Shell sidebar.
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: Zero, epaine, gvanrossum, rhettinger, taleinat, terry.reedy
Priority: normal Keywords: patch

Created on 2019-08-21 06:47 by terry.reedy, last changed 2020-09-16 20:21 by epaine.

Files
File name Uploaded Description Edit
screenshot.png taleinat, 2019-11-01 22:30
screenshot2.png taleinat, 2019-11-12 06:27
Pull Requests
URL Status Linked Edit
PR 15474 open taleinat, 2019-08-24 20:50
Messages (28)
msg350054 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-08-21 06:47
This issue proposes to add a shell-specific sidebar to Shell.  It follows-up #17535, which added a line-number sidebar to module editors and implements the first phase of #37892, which proposes that the single statement editing area of Shell should be strictly just that.

Looking ahead to this issue, #17535 added both a generic Sidebar class and a specific LineNumber subclass.  This issue proposes to add a 3 char wide ShellIO subclass that would mark the beginning of the blocks of text added to the Shell history area. Labels, such as '>>>', 'Out', 'Inp', and 'Err' would be used for the first line of user code input, program stdout, user response to input() (there is only one line), and program stderr (which includes tracebacks).  I am not quite sure what to do with debug notices and warnings from the warning modules.  Maybe use 'Con', maybe use Dbg and Wrn.  I expect to test variations.

As with LineNumber, the font face and size will the the same as in the text.  The labels should use the highlight colors of their text block except that '>>>' should continue getting the 'console' colors.

I thing the initial implementation should not response to clicks.  After experimentation, we might decide that clicking on a label could select whole blocks, but I would like get the essential stuff right first.
msg350400 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-08-24 20:51
See PR GH-15474 with an implementation.
msg355752 - (view) Author: Stephen Paul Chappell (Zero) Date: 2019-10-31 13:58
The documentation for sys.ps1 and sys.ps2 states that they "are only defined if the interpreter is in interactive mode." Since the IDLE shell is meant to be interactive (and to reduce the differences between the shell and running Python directly), would it be helpful if ps1 and ps2 were defined when running IDLE? The shell could then honor their values.

If such a direction was explored, one issue may be that the sidebar could not simply be 3 char wide. The documentation also states that non-strings are evaluated each time they are needed by the interpreter. This allows for such interesting possibilities as shown with the code below but may not be desired functionality for the IDLE shell window.

import sys
from datetime import datetime

class TimePrompt:
    def __init__(self, prefix, suffix):
        self.prefix, self.suffix = prefix, suffix
    def __str__(self):
        return f'{self.prefix}{datetime.now()}{self.suffix}'

sys.ps1, sys.ps2 = TimePrompt('', ' >>> '), TimePrompt('', ' ... ')
msg355755 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-10-31 15:03
Will the current interactive shell continue to be available?
msg355756 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-10-31 15:10
Stehpen, I'm not sure what your use-case is... To me it seems that sys.ps1 (and ps2?) aren't quite relevant any more when using a sidebar, since there are no longer inline prompts.

Raymond, with the current PR, this replaces the inline prompts. Making it toggle-able could be very awkward. Is this a deal-breaker in your opinion?
msg355758 - (view) Author: Stephen Paul Chappell (Zero) Date: 2019-10-31 15:37
Maybe my impression has been false this whole time, but the Python interactive interpreter seems to be very similar to the IDLE shell window. My question is, "Why not make them even more so?" Having IDLE react to sys.ps1 and sys.ps2 opens up the shell to similar use cases as having them defined in the interactive interpreter. My suggestion is not to have them added as text as is usual in a terminal window but to print the values of ps1 and ps2 in the proposed ShellIO subclass. That would make label customization possible in a way that is already documented and standard.
msg355803 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-01 11:07
Stephen, perhaps what you're suggesting is out of context for this issue? This issue is about removing prompts from the shell window's text widget, replacing them with a separate sidebar showing where prompts and user input were. This will make IDLE's shell less similar to the basic command line shell, not more similar.
msg355812 - (view) Author: Stephen Paul Chappell (Zero) Date: 2019-11-01 12:12
Zero:     "not to have them added as text as is usual in a terminal window"
taleinat: "removing prompts from the shell window's text widget"

Zero:     "print the values of ps1 and ps2 in the proposed ShellIO subclass"
taleinat: "separate sidebar showing where prompts and user input were"

We appear to be in agreement.

terry.reedy: "Labels, such as '>>>', 'Out', 'Inp', and 'Err' would be used"
Zero:        "Having IDLE react to sys.ps1 and sys.ps2"

My suggestion is to take those labels terry.reedy talks about from the values of ps1 and ps2 since they are already documented and standard for "the interpreter ... in interactive mode." If psX needs to introduced for other prompts that may be needed ("Maybe use 'Con', maybe use Dbg and Wrn."), it would provide a sensible way to customize those labels as well.
msg355837 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-01 20:05
After the PR, can someone easily  reproduce the interactive prompt sessions in the tutorial and maindocs.  Will it look the same?  Would they be able to build doctest examples in the IDLE shell?

This is marked for 3.7 through 3.9.  Do you intend to backport this PR?
msg355838 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-01 20:34
> After the PR, can someone easily  reproduce the interactive prompt sessions in the tutorial and maindocs.

Yes, except with prompts (both >>> and ...) in the sidebar rather than inline.

> Will it look the same?

No, see above.

> Would they be able to build doctest examples in the IDLE shell?

That's a good thought. It should be rather simple to add a context menu option to "copy for doctest" which would add the prompts to the copied text.

(By the way, how popular are doctests these days? My impression is that they've completely fallen out of style, but I'm sure that could vary between different areas and industries.)
msg355839 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-01 20:38
Stephen (Zero), thanks for the clarification.

FWIW I'm -1 on the ability to customize the prompts in the sidebar. It would complicate the implementation while reducing the straightforwardness and simplicity of IDLE.

Also, anyone who *really* wants to customize them could just change the hard-coded values in the code in their own copy of IDLE. I don't think it's worth the effort to make it more easily configurable.
msg355840 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-01 20:41
> This is marked for 3.7 through 3.9.  Do you intend to backport this PR?

IDLE generally has a different backporting policy: We generally backport all IDLE development to all versions of Python that aren't in security-only or unmaintained status, except for 2.7. Currently, that means 3.7 and above.

There is always room for exceptions, of course.
msg355844 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-01 21:14
> By the way, how popular are doctests these days?

Arguably, Sphinx has made them more popular than ever.  For me, they are essential.  And as long as doctest remains a non-deprecated module, we have to retain support.

For the record, I'm opposed to backporting this to 3.8 and 3.7.  If this PR ends-up killing our teaching workflow, we need something to fallback to.  Also, you don't want to immediately invalidate every video or written tutorial that uses IDLE for demonstrations.

> Raymond, with the current PR, this replaces the inline prompts. 
> Making it toggle-able could be very awkward. Is this a 
> deal-breaker in your opinion?

I can't tell for sure because I'm having a hard time building with Tkinter support, so I can't easily try out the PR.
msg355847 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-01 21:50
As stated in my second opening sentence, this issue "implements the first phase of #37892", which gives the context and specific rationale.

There are 3 separate parts to a Python Shell.

1. Code entry, indentation, display, and copying.  This issue fixes Shell design flaws that have been recognized as such for at least 15 years.  In the process, it makes Shell *more similar* to the standard REPL in important respects.  In particular, by default, compound statements will look the same in Shell as the same code entered into REPL with 4-space indents.  It will also be possible to cut and paste into an editor have it look right.

Raymond: I do not plan to retain a buggy shell mode that in #7676 you called a "major PITA" (msg215288) and "a continual source of frustration for students in my Python courses" that you were "looking forward to it being fixed."

2. Code execution.  IDLE executes syntactically correct user code so that, to the extent possible and sensible, the result is the same as if executed directly with python.  Not an issue here.

3. Everything else.  Alternate shells are intentionally difference from and hopefully improve upon the interactive python.exe REPL.

Paul: The existence of two prompts, sys.ps1 and sys.ps2 in python.exe interactive mode is a kludge needed to interface nested multiline statement Python with *single text line entry* system terminals.  I believe that most beginners never fiddle with them.

IDLE's Shell is a GUI-based python-statement-aware interface.  The python.exe executing user code from shell or editor is never in interactive mode.  For statements entered and executed, '>>>' is quite sufficient to mark code.  The sidebar should be minimal and unobtrusive and 3 columns is sufficient.  So I reject the idea of expanding it.

#37892 mentions the possibility of later adding an expanded full-line prompt for the code entry area, such as "Enter below a complete Python statement."  When we get to that, we can consider an option (for non-beginners) to customize it.
msg355850 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-01 22:30
To help get a feel for what the current PR looks like, I'm attaching a screenshot (taken on Windows 10).
msg355851 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-01 22:50
Thanks.  Can you also show a script running with F5 showing a restart, the output from the script and a subsequent interactive session that shows the inspecting variables.  Also show what an exception and SyntaxError looks like.
msg355859 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-02 02:59
My previous response was to comments up to msg355812.

To answer Raymond's subsequent questions: The user will enter python code the same as before.  Currently, the indents and resulting appearance of the code on Shell screen is different from (and worse, sometimes far worse, than) the appearance of the same code in the REPL.  After this patch *and* a required immediate follow-up to replace tab indents with, by default, 4-space indents, the code on the screen *and in saved sessions* will appear the same as in the REPL (when 4-space indents are used).

In Tal's screenshot, the appearance is better, but the indents are still tabs.  Because of the regression being fixed by #38636, PR 17008, he could not manually switch with alt-T and alt-U.

(The current sidebar patch is also missing output markers.  Also, for my custom dark theme, I colored the sidebar dark blue on light bluish gray and the result, to me, is aesthetically much more pleasing than in the screenshot.)

The improvement is, to me, even more evident in the following pair of examples -- first REPL and improved IDLE, second current IDLE. 

>>> if True:
...     print('true')
... else:
...     print('false')

versus

>>> if True:
        print('true')
else:
        print('false')

#7676 was marked as a bugfix issue and I regard changing examples like the above to be a long-overdue and badly needed fix.  Any version that does not get this fix will also not get followup changes.  I just added msg355853 to #37892 to list some I have thought of.

One of my intended followups is to add multiple ways of saving a shell session.  IDLE can juggle prompts/markers, code, and output to match 'templates' much easier than users can.  I had not thought about doctests, but saving shell interaction in the proper format should be an easy addition.

SyntaxErrors aside, output will be the same as it is in IDLE now, with 'Out', 'Err', and maybe 'Inp' markers ('Imp' for user code input(prompt) lines) planned for the sidebar.

IDLE already handles syntax errors in the code being entered differently from the REPL by replacing the caret line with an error highlight, as in the editor.  I would like it follow the logic of not imitating dumb terminals by putting the cursor at the marked spot, as in the editor, and letting the user directly correct the problem.  Messing up the input with a usually useless traceback and a new prompt and requiring retrieval of the statement with the marker removed and the cursor in the wrong place is a nuisance.  I also don't see any reason to burden the session history with bad code that was never submitted for execution.  (Note that IDLE does not RESTART when code in the editor has syntax errors.)
msg355861 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-02 05:02
Strong +1. I should have done this 20 years ago.
msg355865 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-02 10:35
With the support from Guido and others here and on idle-dev, Terry and I will be moving forward with this change.

However, I take Raymond's concern about this "killing our teaching workflow" seriously. Raymond, let's please work to figure out how to make sure this improves, rather than degrades, the teaching of Python with IDLE. I'd be happy to work on features, updating documentation, outreach, or anything else that this would require.
msg355872 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-02 15:18
Do you still plan to backport to 3.7 and 3.8?
msg355877 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-02 17:27
Yes.
msg355884 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-03 01:31
This feature may be fine or it may not.  I haven't been able to apply the PR and get a working Python 3.9 with Tkinter support, nor have I seen the requested additional screen shots.

Ideally, major changes in appearance or functionality should have significant user testing and feedback.  However, the tone of this thread is that the user feedback is irrelevant and the feature will happen and be backported regardless.

My team has used IDLE 8 hours a day for the last eight years, cumulatively training 25,000+ engineers using IDLE.  We have served as the beta tester and as the canary in the coal mine, reporting every usability issue so that IDLE can be improved.  We have tried to be a voice for users and it is sad to be roundly dismissed.

Other bug reports and feature requests that matter to us don't appear to be getting any traction:

* The squeezer on-by-default degraded our user experience
* Need auto-save shell sessions (we record full day sessions)
* Print speed is slow, giving the appearance of Python being slow
* Need Alt +/- hot keys for changing font size
* Recently lost the strip-trailing-whitespace menu option
* Need auto-strip-trailing-whitespace on save
* Tab completion has been unreliable for several years
* Need an option to run the install-certificates script (otherwise urllib HTTP requests fail with a fresh install of Python from python.org)
* Cntl-A on macOS jumps before the >>> prompt
* Font samples on the Settings tab are useless, distracting, and so large that it can make it impossible to mouse down to the Ok or Apply buttons when the display size is small (constrained by the projector).
* The tab size slider should be a scroll box and the font size scroll-box should be slider (users should almost never change the tab size but almost always want to change the font size).
* The default font size is too small.  For some users, when IDLE opens, the text looks like dust.
* The completions pop-up menu has never been helpful and users have a hard time getting it dismissed (so we usually have people change the completions pop-up time to 10,000 milliseconds)
* The mouse cursor targeting frequently gets missed by about seven lines (my understanding is that this is an upstream Tkinter problem, but it has been around for many years -- we have to close and reopen windows frequently to make this goa away).
* For the past two or three years, we periodically get an unintentional clipboard paste in to the shell window.  This happens during live demos, so the instructors have been accustomed to reflexively type Cntl-Z to undo the paste so as not to interrupt the demo).
* For mac users who have enabled tabs system-wide, opening a new window in IDLE triggers a system log-out (amazing that this is even possible).  My understanding is there isn't anything we can do about this, but it is a really bad out-of-the-box user experience seconds after a fresh install of Python.
* On rare occasion, I teach Python to kids.  A menu option to launch the turtle demo would be helpful.
* Being able to run pip from within IDLE would be a major win, especially for Windows users who are foreign to the command-line and who often do have Python on the PATH.
* The Search Dialog window doesn't stay on top of other windows.  It can have the focus, but be hidden underneath other windows.  The user experience that is there is no key that they can press to continue running their session.
* Sometimes the Search Dialog window becomes unresponsive and there is no way to clear it without turning IDLE off.
* When launching IDLE from the command-line, we see various exceptions and tracebacks for errors in the code (typically along the lines of None-type object doesn't have some attribute).
* Several of these issues have only came-up in the past few years.  IDLE used to be more stable than it is now.
msg355885 - (view) Author: Stephen Paul Chappell (Zero) Date: 2019-11-03 01:59
@rhettinger: The turtle demo is easily accessible through the menus via Help > Turtle Demo.

It is nice to see there are others interested in IDLE's improvement. :-)
msg355887 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-03 03:51
> The turtle demo is easily accessible through the 
> menus via Help > Turtle 

Thanks.  I didn't see that one land.  It is a nice improvement.
msg355888 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-11-03 03:59
Hmm, the turtle demo is barely usable on the mac:

* The "Choose Example from Menu" is a label and not clickable.
* The left pane is clickable but does nothing.
* When code is chosen from the menu the font-size is small.
* The "start" button label is invisible and the "stop" and "clear" buttons are barely visible with yellow-on-white.
* Once started, the "stop" button is invisible and the other two buttons shift to yellow-on-white.
msg355889 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-11-03 06:09
@raymond Please stop spamming this issue with unrelated IDLE issues.
msg356414 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-12 06:27
> Can you also show a script running with F5 showing a restart, the output from the script and a subsequent interactive session that shows the inspecting variables.  Also show what an exception and SyntaxError looks like.

I've attached a screenshot showing everything that Raymond asked for.
msg356415 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-11-12 06:36
Raymond, FWIW, this will fix at least one of the issues you've mentioned: "Cntl-A on macOS jumps before the >>> prompt".
History
Date User Action Args
2020-09-16 20:21:28epainesetnosy: + epaine

assignee: terry.reedy
components: + IDLE
versions: + Python 3.10, - Python 3.7
2019-11-12 06:36:31taleinatsetmessages: + msg356415
2019-11-12 06:27:34taleinatsetfiles: + screenshot2.png

messages: + msg356414
2019-11-03 06:09:07gvanrossumsetmessages: + msg355889
2019-11-03 03:59:56rhettingersetmessages: + msg355888
2019-11-03 03:51:03rhettingersetmessages: + msg355887
2019-11-03 01:59:37Zerosetmessages: + msg355885
2019-11-03 01:31:46rhettingersetnosy: + rhettinger
messages: + msg355884
2019-11-02 18:16:57rhettingersetnosy: - rhettinger
2019-11-02 17:27:53terry.reedysetmessages: + msg355877
2019-11-02 15:18:39rhettingersetmessages: + msg355872
2019-11-02 10:35:17taleinatsetmessages: + msg355865
2019-11-02 05:02:04gvanrossumsetnosy: + gvanrossum
messages: + msg355861
2019-11-02 02:59:23terry.reedysetmessages: + msg355859
2019-11-01 22:50:51rhettingersetmessages: + msg355851
2019-11-01 22:30:10taleinatsetfiles: + screenshot.png

messages: + msg355850
2019-11-01 21:50:54terry.reedysetmessages: + msg355847
2019-11-01 21:14:36rhettingersetmessages: + msg355844
2019-11-01 20:41:04taleinatsetmessages: + msg355840
2019-11-01 20:38:45taleinatsetmessages: + msg355839
2019-11-01 20:34:56taleinatsetmessages: + msg355838
2019-11-01 20:05:11rhettingersetmessages: + msg355837
2019-11-01 12:12:38Zerosetmessages: + msg355812
2019-11-01 11:07:44taleinatsetmessages: + msg355803
2019-10-31 15:37:22Zerosetmessages: + msg355758
2019-10-31 15:10:29taleinatsetmessages: + msg355756
2019-10-31 15:03:59rhettingersetnosy: + rhettinger
messages: + msg355755
2019-10-31 13:58:26Zerosetnosy: + Zero
messages: + msg355752
2019-08-24 20:51:50taleinatsetmessages: + msg350400
2019-08-24 20:50:08taleinatsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request15162
2019-08-21 06:47:13terry.reedycreate