Title: input() and raw_input() do not work correctly with colored prompts
Type: behavior Stage:
Components: Documentation Versions: Python 3.4, Python 3.3, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, ezio.melotti, mic_e, terry.reedy, tshepang
Priority: normal Keywords:

Created on 2013-03-02 15:09 by mic_e, last changed 2013-03-15 20:59 by terry.reedy.

Messages (6)
msg183323 - (view) Author: Michael Enßlin (mic_e) Date: 2013-03-02 15:09
With a prompt that uses ANSI color escape codes, python3 input() and python2 raw_input() behave incorrectly when it comes to the wrapping of the first line - almost certainly due to the wrong string length calculation.
The line breaking occurs k characters early, where k is the number of characters in the ANSI escape sequence.
Even worse, the line break does not switch to the next line, but instead jumps back to the beginning, overwriting the prompt and the previously written input.

How to reproduce:
Call input() with a color-coded string as argument, and type until you would expect the usual line break.


mic@mic-nb ~ $ python3
Python 3.3.0 (default, Dec 22 2012, 21:02:07) 
[GCC 4.7.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> prompt="\x1b[31;1mthis is a bold red prompt> \x1b[m"
>>> len(prompt)
>>> input(prompt)
uvwxyzs a bold red prompt> abcdefghijklmnopqrst 
mic@mic-nb ~ $ python2
Python 2.7.3 (default, Dec 22 2012, 21:14:12) 
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> prompt="\x1b[31;1mthis is a bold red prompt> \x1b[m"
>>> len(prompt)
>>> raw_input(prompt)
uvwxyzs a bold red prompt> abcdefghijklmnopqrst 
mic@mic-nb ~ $ tput cols

I have typed directly after the prompt the string 'abcdefghijklmnopqrstuvwxyz',
so the expected result would be be
this is a bold red prompt> abcdefghijklmnopqrstuvwxyz
with four more characters of space before the line break

Note that the break occurs exactly 8 characters early, which is the total amount of ANSI escape sequence characters.
Also note that the readline module is impored in my .pyrc file.
msg183325 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-03-02 15:43
I can reproduce the issue, but only from the interactive interpreter while using input() directly (Linux/py3).
I tried the following things:

$ ./python -c 'print("\x1b[31;1mthis is a bold red prompt> \x1b[m", end=""); input()'
$ ./python -c 'input("\x1b[31;1mthis is a bold red prompt> \x1b[m");'
>>> print("\x1b[31;1mthis is a bold red prompt> \x1b[m", end=""); input()
>>> input("\x1b[31;1mthis is a bold red prompt> \x1b[m")

In the first 3 cases once I reach the end of the line, the text went on a new line.  In the last case it started writing over the prompt instead of going on a newline, and once it reached the end of line again it went on a newline correctly.
msg183328 - (view) Author: Michael Enßlin (mic_e) Date: 2013-03-02 16:00
The issue might very well be strictly related to GNU readline.

I have both successfully reproduced it in a C program:

#include <stdio.h>
#include <readline/readline.h>
int main() {
        readline("\x1b[31;1mthis is a bold red prompt\x1b[m> ");

gcc -lreadline test.c

and found a fix, hinted at by this stackoverflow post:

Readline uses the characters \x01 and \x02 to mark invisible portions of the prompt, so I am now pre-processing the prompt with this function:

def surround_ansi_escapes(prompt, start = "\x01", end = "\x02"):
        escaped = False
        result = ""

        for c in prompt:
                if c == "\x1b" and not escaped:
                        result += start + c
                        escaped = True
                elif c.isalpha() and escaped:
                        result += c + end
                        escaped = False
                        result += c

        return result

However, in my opionion this fact deserves at least to be mentioned in the readline documentation.
msg183780 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-09 01:07
On Windows, both the command prompt interpreter and IDLE ignore special meanings and print prompt as is. Line wrap to physical next line occurs at 80 chars and movable edge of window respectively.

Michael, in light of your last post, do you still believe a change is needed in the code we distribute, in particular our readline module, whether bugfix or enhancement, or should this be closed? I do not believe that Python itself know how the output device will interpret the characters sent to it.
msg184246 - (view) Author: Michael Enßlin (mic_e) Date: 2013-03-15 18:43
Terry, i guess you are right; it is indeed not the job of python to know how its terminal will print characters; there is a whole lot of issues to consider, such as terminal unicode support, control characters, ansi escape sequences, terminal-specific escape sequences such as terminal title, etc.
I guess that on UNIX, a lot of that information could be taken from terminfo, to provide a method that guesses the length of a text on the terminal that is referenced in the $TERM environment variable.

In fact, I think this is a design problem of readline; readline should print the prompt, and then try to determine its length via the dedicated escape sequence (if it really needs to know the prompt length). In that case, there would be no way to fix this in python - apart from re-implementing. In conclusion, no, I don't believe the python readline code should be modified, maybe an additional, os- and $TERM-dependent method for adding prompt escapes should be added.

However, the issue should definitely be documented in the readline documentation:
importing readline _will_ break input() if the prompt contains non-printable characters that are not escaped by '\x01' and '\x02' characters.

Hence I have changed the bug components to 'Documentation'.
msg184257 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-15 20:59
Please suggest the specific change in a specific place that you would like to see. Message is ok, patch even better.
Date User Action Args
2013-03-15 20:59:42terry.reedysetmessages: + msg184257
2013-03-15 18:43:22mic_esetnosy: + docs@python
messages: + msg184246

assignee: docs@python
components: + Documentation, - Library (Lib)
2013-03-09 01:07:04terry.reedysetnosy: + terry.reedy
messages: + msg183780
2013-03-09 00:12:28tshepangsetnosy: + tshepang
2013-03-02 16:00:01mic_esetmessages: + msg183328
2013-03-02 15:43:13ezio.melottisetnosy: + ezio.melotti

messages: + msg183325
versions: + Python 3.4
2013-03-02 15:09:24mic_ecreate