In the past this was worked around by installing a third-party module:
https://github.com/pyreadline/pyreadline

For Python 3.10, however this module doesn't work.  And pyreadline doesn't seem to be maintained anymore, so it's possible it won't get fixed.
https://github.com/pyreadline/pyreadline/issues/73

Consider the following code:

import cmd
import readline

open("history.txt", "w").write("first line\nsecond line\n")

readline.clear_history()
readline.read_history_file("history.txt")

class MyShell(cmd.Cmd):
    def __init__(self):
        super().__init__()

shell = MyShell()
shell.cmdloop()

This works fine on MacOs Python, also on Linux and on Windows prior to 3.10 with the readline module added.  It won't work going forward.

The Windows cmd implementation clearly uses readline or some compatible lib under the hood, so Python should make it available to developers.

> The Windows cmd implementation clearly uses readline or 
> some compatible lib under the hood

The REPL shell and input() call PyOS_Readline(). If this call isn't hooked (e.g. by the readline module), and stdin is a console file, then it reads a line from the console via ReadConsoleW(). If the console's input stream is configured in line-input mode, which we assume it is, this function provides basic readline-ish support, including a line editor that supports input history and aliases. The history is stored in an in-memory buffer in the console, which doesn't get save and reused across console sessions. There is no public API to get the history contents, and there is no support at all to set it. 

The console/terminal team at Microsoft apparently don't want to do anything with the builtin readline support, which is seen as a legacy feature. To the contrary, I've even seen them suggest that they're evaluating the addition of a new client-side readline library in the native API, which I assume would be similar to PowerShell's PSReadLine (e.g. based on ReadConsoleInputW(), and supporting a classic console mode in addition to emacs and vi modes).

For now, I suggest patching pyreadline for your own use. In the reported issue, I see it's trying to use collections.Callable. Instead it should import collections.abc and use collections.abc.Callable.

Thank you Eryk for the info.

As a bug report, this should be closed as '3rd party'.  As an enhancement request, it needs to be specified more and should perhaps be discussed on python-ideas.

You can take the view that it's not a bug (with some justification), but a few lines in the cmd docs would make all the difference in terms of wasted time.

I have now abandoned my Windows port and suggested users install WSL2 instead which is the easiest way forward for me, but it'd be nice to have known from the start that portions of cmd functionality are not available for Win32 instead of the indirect references via readline.  You could throw us a bone here.

What specific sentences would you like where in which doc.  (Please link as 'cmd doc' is too vague.)

Sorry, you obviously mean
https://docs.python.org/3/library/cmd.html#module-cmd
What to add where still applies.

Guido and Raymond, you are the two active coredevs that have contributed the most lines to cmd module.  What do either of you think?

> You can take the view that it's not a bug (with some justification), 
> but a few lines in the cmd docs would make all the difference in 
> terms of wasted time.

If anything, I think the readline documentation should have a note explaining the situation in Windows. The documentation of the cmd module already makes the readline dependency clear: 

    If the readline module is loaded, input will automatically inherit 
    bash-like history-list editing (e.g. Control-P scrolls back to the
    last command, Control-N forward to the next one, Control-F moves the 
    cursor to the right non-destructively, Control-B moves the cursor to 
    the left non-destructively, etc.).

AFAIK the reason command history works in cmd.py on Windows is that it's built into the terminal program. Or maybe into the operating system.

Thus, the user can use line editing and history, but there is no API (in Python) to interact with these.

I'm sure Steve Dower can explain the exact situation -- it may depend on which Windows version and which terminal program you use (my only recent experience is with winterm on Windows 10).

I agree that (once we sort out what works in what versions of Windows and which terminal programs) we should clarify this in the docs.

> AFAIK the reason command history works in cmd.py on Windows is 
> that it's built into the terminal program. Or maybe into the
> operating system.

As mentioned in msg406800, input editing, history (e.g. up/down arrows, F7 popup, F8 completion), and alias support is implemented by the Windows console host (conhost.exe or openconsole.exe) for ReadFile() and ReadConsole() calls when the input stream is in line-input mode. Currently, it's the same whether a classic console session or a pseudoconsole (headless) session is hosted. When Windows Terminal is used, the overall connection of components looks like Python<->ConDrv (kernel device)<->OpenConsole<->NamedPipe (kernel device)<->Windows Terminal. The headless console's use of Windows Terminal for the user interface doesn't matter to Python's ReadConsoleW() call.

A headless console session always starts with 4 history buffers (one for each attached process) that store up to 50 commands. For a classic console session, the initial number and size of history buffers can be configured in the session properties or defaults. It can always be set dynamically via SetConsoleHistoryInfo(). There's *undocumented* support to get the commands from a history buffer that's associated with the name of an attached process: GetConsoleCommandHistoryLengthW(executable_name) and GetConsoleCommandHistoryW(buffer, buffer_length, executable_name). However, the API provides no function to set the command history. I suppose one could loop over WriteConsoleInputW() and ReadConsoleW() to implement it as a kludge.

Thanks, Eryk, I only read the part of the issue that landed in my inbox (fhe first message and everything after Terry added me to the nosy list). Sorry.

You wrote:

> The console/terminal team at Microsoft apparently don't want to do anything with the builtin readline support, which is seen as a legacy feature.

What does "the builtin readline support" refer to here? Presumably not GNU Readline?

> What does "the builtin readline support" refer to here? 
> Presumably not GNU Readline?

That's referring to the readline(ish) support that's built into the console host for ReadFile() and ReadConsole() calls when the input stream is in line-input mode. I've never seen the console developers speak positively of this feature on their GitHub repo. They've suggested the addition of a native readline API on the client side, like PowerShell's PSReadLine module provides. But who knows when/if that would be released.

Python has the third-party pyreadline module, but it's no longer actively developed. To bring pyreadline into the standard library would be a non-trivial task. OTOH, I assume if Microsoft provided an official readline API, which does all the heavy lifting, that Python could support it in the readline extension module, if the API is basically compatible with libreadline/libedit.

Okay, so that's all hypothetical. It looks like the status quo is not
likely to change, so we should just document it. I wonder if keeely is
interested in submitting a PR for the docs?

Regrettably I cannot submit a PR for the docs because I value my online anonymity and Python submissions require a real name (IIRC), but my suggestion would be pretty simple.

Taking as an example, for termios (https://docs.python.org/3/library/termios.html), we currently have:

This module provides an interface to the POSIX calls for tty I/O control. For a complete description of these calls, see termios(3) Unix manual page. It is only available for those Unix versions that support POSIX termios style tty I/O control configured during installation.


For readline (https://docs.python.org/3/library/readline.html#module-readline) we have:

The readline module defines a number of functions to facilitate completion and reading/writing of history files from the Python interpreter. This module can be used directly, or via the rlcompleter module, which supports completion of Python identifiers at the interactive prompt. Settings made using this module affect the behaviour of both the interpreter’s interactive prompt and the prompts offered by the built-in input() function.

In similar way to the first para of the termios description I would add the following text:  “It is only available on platforms that support the readline functionality, generally POSIX”.


Then perhaps I’d also add to the cmd documentation at https://docs.python.org/3/library/cmd.html

The Cmd class provides a simple framework for writing line-oriented command interpreters. These are often useful for test harnesses, administrative tools, and prototypes that will later be wrapped in a more sophisticated interface.

I would add at the end of that first paragraph:
“Some features will be unavailable on non-POSIX platforms due to the readline requirement”.

Hope this helps, if it's not clear I can provide in diff form if you prefer.

I'm attaching an example usage of cmd + readline to show how you can have context-specific history for sub-shells.  WARNING: WRITES FILES TO CWD! In the event that someone does implement this on Windows it would be nice if this worked.  That doesn't mean less-capable readline support wouldn't also be useful.

thanks.

I would like to make the python community aware that there has recently been a renewed interest in updating and maintaining 'pyreadline', but in a new(ish) repository 'pyreadline3'. 

https://github.com/pyreadline3/pyreadline3

This apparently now works under Py3.10, but there are some minor issues with how pyreadline handles ANSI escape sequences for coloring auto completion etc. This is probably due to the code developed for the older/original Windows cmd/powershell use of 4-bit ANSI colors, no longer being able to handle 256 or RGB. There was a PR for that, but because that PR is aged it seem limited to older 4-bit ANSI's, since it is using a RegEx for stripping the sequences to calculate the actual line length.

[OT. This also bring up the question where the function "rl_expand_prompt()" (handling RL_PROMPT_START_IGNORE and RL_PROMPT_END_IGNORE) was moved, as it can't be found in the cpython codebase. (https://github.com/python/cpython/blob/main/Modules/readline.c) The equivalent HEX for those _does_ work, at least for the prompt, but maybe they are handled on the windows side?]