classification
Title: Change input() to always prompt to stderr
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Daniel.Gonzalez, bhuvan, eric.araujo, ezio.melotti, ggenellina, jason.coombs, martin.panter, mdomingues, serhiy.storchaka, taleinat
Priority: normal Keywords: patch

Created on 2008-01-24 20:21 by skip.montanaro, last changed 2016-02-07 01:20 by martin.panter.

Files
File name Uploaded Description Edit
promptOutputFix.patch mdomingues, 2012-09-24 01:21 A proposed patch to print raw_input prompts to standard out. review
promptOutputFix3.patch mdomingues, 2012-09-24 01:26 Fixes the input prompt issue on the Python3 branch. review
promptOutputFix3.v2.patch martin.panter, 2015-12-05 08:12 review
promptOutputFix3.v3.patch martin.panter, 2016-01-19 03:13 review
Messages (23)
msg61652 - (view) Author: Skip Montanaro (skip.montanaro) * Date: 2008-01-24 20:21
From a thread on python-dev...

http://mail.python.org/pipermail/python-dev/2008-January/076446.html

Mike Kent mike.kent at sage.com
Thu Jan 24 16:33:47 CET 2008

Recently I was trying to debug an old python program who's maintenance I
inherited.  I was using the quick-and-dirty method of putting some 'print
>>sys.stderr' statements in the code, and then running the command with
'2>filename' appended to the end of the command line.  Imagine my surprise
to see that all of the prompt text from the program's raw_input calls were
also disappearing from the screen output, and appearing in the stderr
output routed to the file.

The latest documentation for raw_input states "If the prompt argument is
present, it is written to standard output without a trailing newline."
I posted a question regarding the observed behavior to comp.lang.python
and Gabriel Genellina (thanks Gabriel!) pointed out that despite the
documentation, raw_input was hard-coded to always output its prompt text
to stderr.

This raises two questions:
1. Shouldn't the current documentation be corrected to state that raw_input
writes its prompt to standard error?
2. Is this really the hard-coded behavior we want?  I don't think my
use-case is that odd; in fact, what I find very odd is that the prompt
output is send to stderr.  I mean, I'm printing the prompt for a question,
not some error message. Can there not at least be an optional parameter to
indicate that you want the output sent to stdout rather than stderr?

... after a few responses ...

Guido van Rossum guido at python.org
Thu Jan 24 21:09:12 CET 2008

On Jan 24, 2008 11:41 AM, Mike Kent <mike.kent at sage.com> wrote:
...
> Interesting point about whether GNU readline is installed.  My setup
is RedHat
> Linux, with Python 2.5 that I built and installed myself.  GNU
readline is not,
> in fact, installed.  If you look at Python2.5/Parser/myreadline.c,
function
> PyOS_StdioReadline, line 125, you will see that prompt output is being
sent to
> stderr.  As best as my Python-fu can determine, this is the code used
to output
> a raw_input prompt (thanks again to Gabriel Genellina for pointing me
in the
> right direction.)
>
> It's entirely likely that the difference in what I am seeing and what
you guys
> are seeing is caused by my not having GNU readline installed. 
Nevertheless,
> the behavior without it seems wrong, and is certainly different from the
> documentation.

Agreed.
msg61655 - (view) Author: Gabriel Genellina (ggenellina) Date: 2008-01-24 22:31
GNU readline is configured as to prompt the user using standard output, 
and read input from standard input; if this is the desired behavior it 
would be easy to provide a simple patch so input/raw_input behave that 
way even when readline is not used.
msg116936 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-09-20 13:53
Any *NIX gurus who can sort this one?
msg154049 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-23 04:22
From reading the code for raw_input in 2.7 or input in 3.3 (Python/bltinmodule.c:1573), it looks to me that stdout is used, which would mean this issue is fixed.  However I browsed the file history and could not find the commit that changed this, and my C skills are limited, so I’m adding Ezio to nosy to have another pair of eyes confirm.
msg171087 - (view) Author: Michael Domingues (mdomingues) Date: 2012-09-24 01:21
The code that dictates this behavior is in /Parser/myreadline.c and has not been rectified yet in either Python 2.7 (http://hg.python.org/cpython/file/bfdf366a779a/Parser/myreadline.c#l107) or the default branch (http://hg.python.org/cpython/file/c64dec45d46f/Parser/myreadline.c#l111). Specifically, within these functions, references to standard error should actually be references to standard out.

The attached file is a proposed patch for this bug on the 2.7 branch, bringing interpreter behavior into accordance with the Python documentation (http://docs.python.org/library/functions.html#raw_input), which states that the prompt is written to standard out, as opposed to standard error.
msg171088 - (view) Author: Michael Domingues (mdomingues) Date: 2012-09-24 01:26
Also uploading a patch for the Python3.2 branch.
msg177966 - (view) Author: Daniel Gonzalez (Daniel.Gonzalez) Date: 2012-12-23 09:33
Please see this stackoverflow thread where more information is given about this issue:

http://stackoverflow.com/questions/14009714/strange-redirection-effect-with-raw-input
msg223493 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-07-20 01:35
I experimented with various redirections to /dev/null, files, and other terminal windows on Linux. Current behaviour I am seeing seems to be something like this:

* Prefers prompting to stderr if both stdout and stderr are terminals
* Prefers prompting to stdout if neither are terminals
* Prompts to the non-terminal if only one of stderr and stdout is a terminal. Surely this one should be the other way around if there is going to be a preference at all?
msg241693 - (view) Author: Bhuvan Arumugam (bhuvan) Date: 2015-04-21 02:26
For the record, this bug is still open. 

The proposed patch is not merged in any of branches.

The prompt for raw_input in all versions, go to stderr.
msg246392 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2015-07-07 07:11
See also issue #24402: input() uses sys.__stdout__ instead of sys.stdout for prompt
msg255080 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-11-22 01:13
The input() implementation is a bit like this:

def input(prompt):
    if stdin and stdout are the original file descriptors, and are terminals:
        return PyOS_Readline(sys.stdin, sys.stdout, prompt)
    else:
        sys.stdout.write(prompt)  # Writes to stdout
        return sys.stdin.readline()

def PyOS_StdioReadline(stdin, stdout, prompt):
    '''Default implementation of PyOS_Readline()'''
    sys.stderr.write(prompt)  # Writes to stderr
    return stdin.readline()

def call_readline(stdin, stdout, prompt):
    '''Implementation of PyOS_Readline() in the "readline" module'''
    rl_instream = stdin
    rl_outstream = stdout  # Readline writes to stdout
    return readline_until_enter_or_signal(prompt)

It looks like PyOS_StdioReadline() has always written to stderr. The stdin and stdout parameters of PyOS_Readline() were added later, in revision dc4a0336a2a3.

I think the changes to myreadline.c will also affect the interactive interpreter prompt. But we have Issue 12869 open to change that to stdout, so maybe the change is okay.

Since input() should no longer depend on any instance of stderr, perhaps the check for “lost sys.stderr” should also be removed.

It may be worth applying any changes in myreadline.c to the independent version in pgenmain.c as well, just for consistency.
msg255779 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2015-12-02 21:10
+1 to applying this patch. After reviewing this and Issue 12869, I don't see any substantial objections or concerns. The status is "test needed". Is a test really needed? My instinct that simply aligning the implementation with the docs is sufficient.
msg255796 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-02 23:40
“Test needed” is meant to mean someone needs help producing the problem, but people also seem use it to request a refined test case for the test suite.

A test case is always nice, although in this case it is a bit tricky. I can try to knock one up use the existing infrastructure in test_builtin.PtyTests. A similar test case could probably be made for the interactive interpreter (Issue 12869), but might be more involved, and I don’t think there is any existing code to copy from.
msg255933 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-12-05 08:12
Here is an updated patch for Python 3.

I did not remove the “lost sys.stderr” check I mentioned earlier, because the implementation still needs it to call flush().

Changes compared to Michael’s patch:

* Added a test for input() using a pseudoterminal and subprocess.Popen
* Write to the passed-in sys_stdout parameter, not the global stdout
* Continue to call fflush(stderr), to avoid regressions with buffered stderr messages
* Updated /Parser/pgenmain.c to use sys_stdout

I also had to update test_cmd_line_script, which expected the prompt to be on stderr. This made me wonder if it is a good idea to change where the interpreter prompt (>>>) goes in a bugfix release. AFAIK it is not documented, and it could potentially break other things that use the interactive interpreter. What do people think? A way to avoid this might be to pass stderr as the sys_stdout parameter.

Also, it would be awesome if someone could try my new test_builtins test case on BSD or OS X. I only tested it with Linux. The last time I messed with pseudoterminals like this I caused the tests to hang on BSD buildbots.
msg255955 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2015-12-05 14:29
The entire test suite passes with the v2 patch on my OSX 10.10.
msg258571 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-19 03:13
Tal: thanks for testing.

This v3 patch changes the interactive interpreter to pass stderr as the sys_stdout parameter. This means we should maintain compatibility with the interpreter prompt going to stderr, but still fix input().
msg258576 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-19 09:58
Unix shell builtin command "read" outputs prompt to stderr. In bash that uses readline and in dash that doesn't use readline. Looks as this is standard behavior.
msg258578 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-19 10:36
The way I see it, input() is mainly geared for prompting to stdout, and it is just one aspect that strangely uses stderr:

* Documentation says stdout
* Stdout is checked if it is a terminal and not redirected
* Gnu Readline is configured for stdout
* The fallback for non-terminals uses stdout

Arguments for using stderr:

* Consistency with Unix shell
* Consistency with the Python interactive interpreter prompt

Maybe it is more ideal to use stderr (I have no idea). But I think that would be a more drastic change.
msg259608 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-05 01:19
Serhiy, was your comment an objection to changing away from stderr, or was that just an observation that Python’s design is inconsistent with the rest of the world?
msg259635 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-05 06:38
This is rather an objection.

If Gnu Readline is configured for stdout, why bash outputs to stderr? We should investigate what exactly do bash and other popular programs with readline, and implement this in Python.

Changing the documentation usually is a less drastic change than changing behavior.
msg259636 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-05 07:59
Okay, I see. To clarify, it is Python that sets up Gnu Readline for stdout: <https://hg.python.org/cpython/file/v3.5.1/Python/bltinmodule.c#l1941>. The problem is whichever way we go, we will have to change some part of the behaviour to make it internally consistent. I think my patch is the minimal change required.
msg259643 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-02-05 08:53
I think we first should fix 3.6 in correct way, and then see what we can backport to other branches without breaking too much. Or left them as is. For example the output of --version was changed from strerr to stdout (issue18338, issue18920) in default branch, but this was not backported.
msg259754 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-07 01:20
This proposal is starting to sound reasonable to me. Changing the title to reflect the new direction.
History
Date User Action Args
2016-02-07 01:20:20martin.pantersettitle: raw_input behavior incorrect if readline not enabled -> Change input() to always prompt to stderr
stage: patch review -> needs patch
messages: + msg259754
versions: - Python 2.7, Python 3.5
2016-02-05 08:53:40serhiy.storchakasetmessages: + msg259643
2016-02-05 07:59:54martin.pantersetmessages: - msg259637
2016-02-05 07:59:36martin.pantersetmessages: + msg259637
2016-02-05 07:59:35martin.pantersetmessages: + msg259636
2016-02-05 06:38:28serhiy.storchakasetmessages: + msg259635
2016-02-05 01:19:38martin.pantersetmessages: + msg259608
2016-01-19 10:36:31martin.pantersetmessages: + msg258578
2016-01-19 09:58:29serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg258576
2016-01-19 03:13:16martin.pantersetfiles: + promptOutputFix3.v3.patch

messages: + msg258571
versions: - Python 3.4
2016-01-19 02:50:54martin.panterlinkissue12869 superseder
2016-01-19 00:50:27martin.panterlinkissue13886 dependencies
2015-12-05 14:29:47taleinatsetmessages: + msg255955
2015-12-05 08:12:40martin.pantersetfiles: + promptOutputFix3.v2.patch

messages: + msg255933
stage: test needed -> patch review
2015-12-02 23:40:03martin.pantersetmessages: + msg255796
2015-12-02 21:10:04jason.coombssetmessages: + msg255779
2015-12-02 20:00:52jason.coombssetversions: + Python 3.5, Python 3.6
2015-12-02 20:00:42jason.coombssetnosy: + jason.coombs
2015-11-22 01:13:22martin.pantersetmessages: + msg255080
2015-11-21 23:09:28martin.panterlinkissue25692 superseder
2015-07-07 07:11:05taleinatsetnosy: + taleinat
messages: + msg246392
2015-04-21 02:26:54bhuvansetnosy: + bhuvan
messages: + msg241693
2014-07-20 01:35:18martin.pantersetnosy: + martin.panter

messages: + msg223493
versions: + Python 3.4
2014-02-03 19:04:58BreamoreBoysetnosy: - BreamoreBoy
2012-12-23 09:33:18Daniel.Gonzalezsetnosy: + Daniel.Gonzalez
messages: + msg177966
2012-09-24 01:26:28mdominguessetfiles: + promptOutputFix3.patch

messages: + msg171088
2012-09-24 01:21:31mdominguessetfiles: + promptOutputFix.patch

nosy: + mdomingues
messages: + msg171087

keywords: + patch
2012-02-23 04:22:36eric.araujosetnosy: + ezio.melotti, eric.araujo
messages: + msg154049
2010-09-20 13:53:20BreamoreBoysetnosy: + BreamoreBoy

messages: + msg116936
versions: + Python 2.7, - Python 2.6
2010-05-20 20:36:08skip.montanarosetnosy: - skip.montanaro
2010-01-29 04:42:52brian.curtinsetstage: test needed
versions: - Python 2.5
2008-01-24 22:32:00ggenellinasetnosy: + ggenellina
messages: + msg61655
2008-01-24 20:33:44christian.heimessetpriority: normal
2008-01-24 20:21:28skip.montanarosettype: behavior
2008-01-24 20:21:17skip.montanarocreate