classification
Title: Interactive interpreter doesn't flush stderr prompty
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, David.Edelsohn, dmalcolm, eryksun, marczellm, ncoghlan, pitrou, python-dev, skrah, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2014-05-04 08:28 by marczellm, last changed 2014-06-12 09:30 by berker.peksag. This issue is now closed.

Files
File name Uploaded Description Edit
flush_stderr.patch pitrou, 2014-05-10 14:02
Messages (19)
msg217861 - (view) Author: Márton Marczell (marczellm) Date: 2014-05-04 08:28
When I run a Python 3.3.4 prompt inside Emacs 24.3 on Windows 7, correct commands are evaluated immediately, but incorrect ones are delayed (I have to press Enter one more time), as seen below:

    >>> 1
    1
    >>> nonsense
    >>>
    Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    NameError: name 'nonsense' is not defined

Python 2 does not do this. I've filed an Emacs bug report (http://debbugs.gnu.org/cgi/bugreport.cgi?bug=17304) and got the response to try this on the command line (where cat.exe is from an MSYS installation):

    python 2>&1 | cat.exe

and it behaves the same way as in Emacs.
msg218218 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-10 13:15
This can be reproduced as easily under Linux:

$ ./python 2>&1 | cat
Python 3.5.0a0 (default:17689e43839a+39f2a78f4357+, May  9 2014, 00:30:19) 
[GCC 4.8.1] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 5
5
>>> 1/0
>>> nonsense
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ZeroDivisionError: division by zero
>>> 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'nonsense' is not defined
>>>
msg218219 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-10 14:02
Attached patch fixes it under Linux, and adds tests.
I'd be glad if someone could give it a run under Windows.
msg218252 - (view) Author: eryksun (eryksun) Date: 2014-05-11 05:21
The patch fixes `python 2>&1 | cat.exe` for me on Windows, where cat.exe is from GnuWin32. The tests also pass. It looks like failing would entail hanging until a timeout.
msg218261 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 11:45
New changeset ab3e012c45d0 by Antoine Pitrou in branch '3.4':
Issue #21425: Fix flushing of standard streams in the interactive interpreter.
http://hg.python.org/cpython/rev/ab3e012c45d0

New changeset d1c0cf44160c by Antoine Pitrou in branch 'default':
Issue #21425: Fix flushing of standard streams in the interactive interpreter.
http://hg.python.org/cpython/rev/d1c0cf44160c
msg218262 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 11:46
Thanks for the report. This should now be fixed in 3.4 and 3.5.
msg218264 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 13:20
Apparently the new tests hang on the "Fedora without threads" buildbot.
Stefan, could you take a look?

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.4/builds/123
msg218265 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 13:52
Also on the PowerLinux buildbot:
http://buildbot.python.org/all/builders/PPC64%20PowerLinux%203.4/builds/113/
msg218266 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 14:04
Judging by the faulthandler traceback in the PowerLinux build:

Timeout (1:00:00)!
Thread 0x0000008074e46670 (most recent call first):
  File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 192 in interactive_python
  File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/contextlib.py", line 59 in __enter__
  File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 205 in check_repl_stderr_flush
  File "/home/shager/cpython-buildarea/3.4.edelsohn-powerlinux-ppc64/build/Lib/test/test_cmd_line_script.py", line 220 in test_repl_stderr_flush
[snip]

... the test is actually stuck trying to read the interactive prompt from stderr:

            # Drain stderr until prompt
            while True:
                data = stderr.read(4)
                if data == b">>> ":
                    break
                stderr.readline()            <-- HERE
msg218267 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 14:09
New changeset ffae7aad9dfa by Antoine Pitrou in branch 'default':
Add debugging output for #21425
http://hg.python.org/cpython/rev/ffae7aad9dfa
msg218269 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 14:44
So, it seems like under Fedora (or PowerLinux), the Python interactive prompt produces different byte contents on stderr:

[ 21/389] test_cmd_line_script
repl stderr[:4]: b'Pyth'
repl stderr[:4]: b'[GCC'
repl stderr[:4]: b'Type'
repl stderr[:4]: b'\x1b[?1'

I wonder if it's because of GNU readline?
msg218271 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 14:54
This looks like a well-known issue on Fedora (and Red Hat?):
http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html

Apparently using XTERM=vt100 would workaround the issue.
(I can't reproduce under Ubuntu)
msg218272 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 14:59
New changeset e57718ac8ff8 by Antoine Pitrou in branch 'default':
Try workaround for test issues in #21425
http://hg.python.org/cpython/rev/e57718ac8ff8
msg218274 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 15:15
According to https://bugzilla.redhat.com/show_bug.cgi?id=880393, this has actually been reported to us as issue19884.
msg218276 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 15:29
Workarounds seem to work.
msg218277 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 15:30
New changeset 58e2116576cf by Antoine Pitrou in branch '3.4':
Try workaround for test issues in #21425
http://hg.python.org/cpython/rev/58e2116576cf
msg218280 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-05-11 17:08
Workaround actually broke tests under shared library builds, because LD_LIBRARY_PATH isn't forwarded anymore. Will try another fix :-(
msg218281 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 17:13
New changeset 5a1b2566d68e by Antoine Pitrou in branch 'default':
Try to fix issue #21425 workaround for shared library builds
http://hg.python.org/cpython/rev/5a1b2566d68e
msg218282 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-11 17:39
New changeset 974c0718c7e0 by Antoine Pitrou in branch '3.4':
Try to fix issue #21425 workaround for shared library builds
http://hg.python.org/cpython/rev/974c0718c7e0
History
Date User Action Args
2014-06-12 09:30:01berker.peksagsetstatus: open -> closed
2014-05-11 17:39:23python-devsetmessages: + msg218282
2014-05-11 17:13:49python-devsetmessages: + msg218281
2014-05-11 17:08:16pitrousetstatus: closed -> open

messages: + msg218280
2014-05-11 15:31:13pitrousetstatus: open -> closed
2014-05-11 15:30:47python-devsetmessages: + msg218277
2014-05-11 15:29:11pitrousetmessages: + msg218276
2014-05-11 15:15:25pitrousetmessages: + msg218274
2014-05-11 14:59:21python-devsetmessages: + msg218272
2014-05-11 14:54:11pitrousetnosy: + dmalcolm
messages: + msg218271
2014-05-11 14:44:05pitrousetmessages: + msg218269
2014-05-11 14:09:24python-devsetmessages: + msg218267
2014-05-11 14:04:32pitrousetmessages: + msg218266
2014-05-11 13:52:44pitrousetnosy: + David.Edelsohn
messages: + msg218265
2014-05-11 13:20:56pitrousetstatus: closed -> open
nosy: + skrah
messages: + msg218264

2014-05-11 11:46:02pitrousetstatus: open -> closed
resolution: fixed
messages: + msg218262

stage: patch review -> resolved
2014-05-11 11:45:20python-devsetnosy: + python-dev
messages: + msg218261
2014-05-11 11:35:16Arfreversetnosy: + Arfrever
2014-05-11 05:21:17eryksunsetnosy: + eryksun
messages: + msg218252
2014-05-10 14:02:07pitrousetfiles: + flush_stderr.patch

nosy: + ncoghlan, tim.golden, zach.ware
messages: + msg218219

keywords: + patch
stage: needs patch -> patch review
2014-05-10 13:15:56pitrousettitle: Python 3 pipe handling breaks python mode in emacs on Windows -> Interactive interpreter doesn't flush stderr prompty
components: + Interpreter Core, - Windows

nosy: + pitrou
versions: + Python 3.5
messages: + msg218218
stage: needs patch
2014-05-10 01:03:08terry.reedysetversions: + Python 3.4, - Python 3.3
2014-05-04 08:28:40marczellmcreate