Title: Slowly introduce a subset of Jupyter console (IPython) features into CPython command line interactive mode
Type: enhancement Stage: resolved
Components: Interpreter Core Versions: Python 3.9
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: Marco Sulla, eryksun, steven.daprano, terry.reedy
Priority: low Keywords:

Created on 2019-11-08 16:38 by Marco Sulla, last changed 2020-03-04 04:45 by terry.reedy. This issue is now closed.

Messages (12)
msg356248 - (view) Author: Marco Sulla (Marco Sulla) * Date: 2019-11-08 16:38
Sometimes I’m lazy and I would test code copy - pasted from internet or from other sources directly in the interpreter, in interactive mode. But if the code contains completely blank lines, the copy - paste fails. For example:

def f():


does not work, but

def f():

yes. Notice that in a script the first code block is perfectly valid and works.

This does not happen with Jupiter console, aka IPython. Jupiter implements bracketed paste mode, so it distinguish between normal input and pasted input.

Jupyter offers also:
- autoindent
- save code blocks in one history entry: this way, if you write a function, for example, and you press the up key, the whole function will be loaded, and not its last line.
- auto-reloading of modules. It should be disabled by default and enabled by a flag, and could auto-reload a module if its code changes.
- save code to console. All the code written in the current interactive session could be saved to the clipboard. It could be triggered by F12.
- history: it could be a new built-in function. if called without parameters, it could show the history, with lines numbered. If called with a number, it will paste the corresponding history line to the console
- pretty printing and source inspection. IMHO pprint.pprint() and inspect.getsource() are so useful in interactive mode that could be added to builtins.
- syntax coloring. It should be disabled by default, and could be enabled by a flag or a config.
- bracket matching. See above.

I think that implementing all of this in CPython is really hard. I suppose that maybe many things are not possible for compatibility between platforms, or can't be available everywhere, like syntax coloring.
msg356283 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-09 02:41
In interactive mode, python.exe interacts with a console/(dumb terminal) through the std streams using \n as a special character  It gets input from stdin, send output to stdout or errors to stderr.  The terminal, not python, handles line editing and history retrieval.  Once a statement is entered and executed, python has no memory of it.

On Linux, one can use readline and ncurses modules for somewhat enhances interaction.

IPython is GUI-based.  Python already come with a GUI-based IDE, IDLE, which has many of the features you list - autoindent, statement history, save, line numbers in the editor, syntax coloring, and some source inspection.  Code with blank lines within statement can be pasted into an editor window and run either with or without clearing the shell workspace.

There are other alternatives with similar features, but this is not the place to discuss them.  The point is that there is no need to completely rewrite current text-based interactive mode.
msg356284 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-11-09 05:00
Terry, I think you were extremely over-eager, almost aggressively so, to close this feature request, especially since your reasons given are rather bogus: IPython isn't based on a GUI, it works in a text mode console too, including on Windows.

You say "there is no need to completely rewrite current text-based interactive mode". You are probably right: there probably is *no need* to completely rewrite the current implementation to add at least some, if not all, of the requested features.

For example, I would be shocked if it wasn't absolutely trivial for the current implementation to add auto-indenting following a colon. That feature alone would be a win for usability.

Given that Brett already said that the main obstacle to this feature request was lack of somebody interested and able to do the work (as opposed to a policy that we want the default REPL to be weak and unfriendly), I think you were premature in closing this so quickly. It's not like it has been languishing for years.
msg356285 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-11-09 05:11
Marco: there's no need for these to be "slowly" introduced. If the features are worth having in the default REPL, they're worth having as soon as possible, without us artificially slowing the process down. It will be hard enough to get somebody willing and able to do the work without telling them to dribble the features out slowly as well. Trust me on this, the hard part of Python development is getting feature requests implemented *at all*, not that they come too quickly!

You might like to try building these features on top of the pure-Python interactive REPL:

or perhaps try adding them to IDLE.

If and when you have something positive to show, you could try re-opening this task with a concrete proof-of-concept using the code module, or perhaps a PR for IDLE.
msg356322 - (view) Author: Marco Sulla (Marco Sulla) * Date: 2019-11-10 10:13
Well, maybe too much feature requests in a single report. I'll report them separately, with more rationale.
msg356323 - (view) Author: Marco Sulla (Marco Sulla) * Date: 2019-11-10 10:22
Steven: currently I'm developing `frozendict` as part of CPython. About IDLE, IDLE can't be used on a server without a GUI. Furthermore, I *really* hope that IDLE is simply a GUI wrapper of REPL, with some additional features.
msg356327 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-10 13:41
> For example, I would be shocked if it wasn't absolutely trivial 
> for the current implementation to add auto-indenting following 
> a colon. That feature alone would be a win for usability.

That would be a non-trivial change in Windows. I think it's at least possible using the high-level console API. It could be implemented with the pInputControl parameter of ReadConsoleW in combination with WriteConsoleW. This is how the CMD shell implements tab completion for file paths. That said, many of the proposed UI enhancements cannot be implemented in Windows using the high-level console API. 

IPython used to depend on readline. (5.0 switched to prompt_toolkit instead.) In Windows this was via the pyreadline package, which uses the low-level console API via ctypes. pyreadline is apparently abandoned (last updated in 2015). Maybe CPython could incorporate a fork of pyreadline that fixes bugs (Unicode support in particular) and updates it to use a C extension instead of ctypes.
msg356330 - (view) Author: Marco Sulla (Marco Sulla) * Date: 2019-11-10 16:25
@Eryk: why a C extension apart and not a patch to `readline`?
msg356345 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-11-11 03:04
Stephen, I think *you* were the one over-anxious to be dismissive.  In the title Marco refers to "Jupyter console (IPython)" features and in his opening, to "Jupyter console, aka IPython".  Jupyter Console is, I read, QT based.  IPython/Jupyter Notebooks are GUI-based also.  

However, I agree that single process terminal IPython, such as illustrated near the top of
is the better reference for comparison.  As pictured, it requires a color terminal with full screen edit.

An important constraint is that instead of users talking to the UI program with menu clicks, they must use "‘magic’ commands" entered after the '>>>' prompt instead of code.  Guido specifically vetoed the idea of allowing these into IDLE, so he must not want them in the REPL either.

Skipping the rest of your post, I will just restate why I closed this issue.

1. It introduces too many features not directly related.  The existing unix-only completions uses two modules.  I suspect some of the other features would also need new modules.  (But Marco, please don't rush to immediately open 8 new issues.)

Furthest removed, no new module needed: Adding builtins that can be used in REPL as it is. I suspect the answer to that proposal would be to use a PYTHONSTARTUP module with code such as "import pprint as _; pprint = _.pprint"

2. It introduces policy issues that might require a policy change, and that I think should better be discussed on, say, pydev.  Steven and I have obviously gotten different policy impressions from Brett and Guido respectively. I will try to ask on pydev what the current REPL feature policy is, and what people think it should be.  For instance, how far do we want to go is adding features that only work on a subset of systems?

3. I believe that some of the concrete proposals have even more implementation problems than Marco himself acknowledged.  Difficult does not mean prohibited, but coredevs are mostly not interested in working on UIs and it has been Guido's stated-to-me policy that 'advanced' UI/IDE features (not defined) should be left to other projects.

One communication problem is that once python is running in interactive mode, the only thing the user can send to it is lines of Python code.  Consider pasting multiline code with blank lines.  The console feeds pasted lines *1 at a time* to interactive Python, the same as if the user typed them.  So Python does not know if they were typed or pasted.  Nor does it know that there might be a next line already waiting, and if so, what it is (dedented?).  If it does not execute on receiving '\n', when will it?

There are also communication issues in the other direction.  When REPL sends a prompt, everything up to and including a prompt is somehow marked read-only.  But autoindents must be left erasable so a user can dedent.

If that can be solved, including on Windows, I would like IDLE's autoindent code moved to an stdlib module (and polished, if need be) to be used by REPL, IDLE, and anyone else who wishes.  The main function would map a sequence of lines to a PEP-8 compliant indent for the next line.

Syntax-coloring opens a much bigger can of worms.  It requires full screen editing to overwrite existing input.  It also needs to be configurable as part of a configuration system.  IPython requires the user to write "a dictionary mapping Pygments token types to strings defining the style." in a python file with other needed code in a place where IPython can find it.  IDLE lets one modify an existing scheme by clicking on an element in sample code and then a new color and let IDLE handle the storage.

Marco, you said "I *really* hope that IDLE is simply a GUI wrapper of REPL".  Nope.  In a single process, python cannot simultaneous execute a file in batch mode and input lines in interactive mode.  IDLE uses "exec(code, simulated_main_namespace)" and I suspect other simulated shells do something similar.  An advantage is that 'code' can represent complete multiline statements or even an entire module.
msg356358 - (view) Author: Marco Sulla (Marco Sulla) * Date: 2019-11-11 11:41

> Jupyter Console is, I read, QT based

Nope. It's shell based by default. You can open it also as a QT app, like IDLE, but by default `jupyter console` is via terminal.

> they must use "‘magic’ commands" entered after the '>>>' prompt 
> instead of code.  Guido specifically vetoed the idea

Indeed I'm against too, and I wrote it. And if you read my proposals, I do not suggest any magic word

> the answer to that proposal would be to use a PYTHONSTARTUP module 
> with code such as "import pprint as _; pprint = _.pprint"

I know this, but it should be the default behaviour, IMHO. I mean, you can invoke `help()` in REPL but also in a `.py`. It makes no sense, but you can do it and you have not to import a separate module before.

> The console feeds pasted lines *1 at a time* to interactive Python

This is fixed by many terminal editors, like `vi`, with bracketed paste mode, as I wrote.

> When REPL sends a prompt, everything up to and including a prompt is 
> somehow marked read-only.

A workaround could be simulate input by user. Ugly but effective.

> Syntax-coloring [...] requires full screen editing


> [Syntax-coloring] also needs to be configurable

This could be IMHO delayed, or not implemented at all. If you don't like the colors, you can always not use it :D It will suffice that the colors will be as much as possible readable also by the majority of color-blind person.
msg363325 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-03-04 04:34
Not surprisingly, I agree with Steven that enhancing IDLE may be a better use of core developer time.  It and Tkinter (and turtle) are options for the Windows and macOS (and some non-PSF) installers and easily installed  on Linux.  IDLE has most of the requested features, and where it does not, I will consider well-specified enhancements.  

1. Pasting a compound statement with embedded blank lines.

>>> def f():
	a = 1  # following line has no spaces or tabs

	print(a, b)

>>> f()
1, 3

Pasting multiple statements, separated or not by blank lines, and pasting non-BMP unicode, are separate issues.

2. Auto-indent: after ':' (see example above) and for syntactic sequences.  (Currently, the latter only works in editors because '>>> ' prompts and tab indents interfere.  There is a issue with PR that will allow this to be fixed.)

strings = ['a',
           'b',  # autoindent
ardvark = fribble(a,
                  b,  # autoindent

3. Code blocks are one history entry.  Yes, where 'code block' means one complete statement, possibly multiple lines, as submitted for execution.  The example in 1. can be recalled as submitted, with the blank line.

4. Save interactive session.  Yes.  I intend to add an option to only save code, without prompts or output, so the code can be pasted into an editor and edited and run.  Or maybe the code should be loaded directly into an editor.

5. Add history() builtin.  I believe that on linux this is redundant with existing enhanced history mechanisms, which include storing history across sessions in a 'standard' place.  I believe my idea for 4. covers must of the use cases.

6. Add a. pprint and b. getsource to builtins.  I think this would be rejected.  I seldom use either of those two functions, but constantly import other things. So I would want other things added.  Startup files are the mechanism for personally customizing one's shell.  Simple imports serve for ad hoc customizations. Note that getsource(name) needs to be passed to print() to be readable.

As for getting source: IDLE displays a popup signature for any function if one is available.  Given a traceback or grep output, one can open any source-based module in an editor at the line indicated.  Open Module lets one easily open any importable source based module with only the import name, not the full path.  For long source chunks, a separate window is really much better than printed in the shell.  A possible IDLE enhancement would be the ability to add an object name and to jump to its definition.

7. Syntax coloring.  Yes, with full customization.

8. Bracket matching for(), {}, and []. Yes, with a couple of options.
msg363326 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-03-04 04:45
>  why a C extension apart and not a patch to `readline`?

Search "python ctypes vs c extension".  Multiple SO answers.  I believe speed and language competancy are big factors.  I don't think and stdlib modules use ctypes.
Date User Action Args
2020-03-04 04:45:16terry.reedysetmessages: + msg363326
2020-03-04 04:34:33terry.reedysetmessages: + msg363325
2019-11-11 11:41:54Marco Sullasetmessages: + msg356358
2019-11-11 03:04:15terry.reedysetmessages: + msg356345
2019-11-10 16:25:58Marco Sullasetmessages: + msg356330
2019-11-10 13:41:04eryksunsetnosy: + eryksun
messages: + msg356327
2019-11-10 10:22:44Marco Sullasetmessages: + msg356323
2019-11-10 10:13:04Marco Sullasetmessages: + msg356322
2019-11-09 05:11:44steven.dapranosetmessages: + msg356285
2019-11-09 05:00:20steven.dapranosetnosy: + steven.daprano
messages: + msg356284
2019-11-09 02:41:44terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg356283

resolution: rejected
stage: resolved
2019-11-08 18:23:06serhiy.storchakasetpriority: normal -> low
type: enhancement
components: + Interpreter Core, - Demos and Tools
2019-11-08 16:38:28Marco Sullacreate