classification
Title: python3.0 -u: unbuffered stdout
Type: behavior Stage: resolved
Components: Versions: Python 3.0, Python 3.1, Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Joe.Borg, benjamin.peterson, fabioz, georg.brandl, mark.dickinson, pitrou, vstinner
Priority: release blocker Keywords: patch

Created on 2008-12-20 13:47 by vstinner, last changed 2013-08-26 16:45 by Joe.Borg. This issue is now closed.

Files
File name Uploaded Description Edit
unbuffered-stdin.patch pitrou, 2009-01-19 17:15
Messages (26)
msg78102 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-20 13:47
I like and I need an "unbuffered" standard output which was provided 
by -u command line option (or PYTHONUNBUFFERED environment variable). 
Current status of -u option in Python3: the option exists and change 
the buffer size (disable buffering) of the stdin, stdout and stderr 
file descriptors.

The problem is in initstdio() which creates files with buffering=-1 
(default buffer) instead of buffering=0 (no buffering) or buffering=1 
(line buffer). But open() enable line buffering of TextIOWrapper is 
buffering=-1 and the raw file is a tty.

Example with py3k trunk:
------------
$ ./python
>>> import sys; sys.stdout.line_buffering
True
$ ./python |cat
>>> import sys; sys.stdout.line_buffering
False
------------

I would like line buffering when stdout is redirected to a pipe and -u 
option is used. initstdio() have to be changed to choose buffering 
option. So it's something like:

Index: Python/pythonrun.c
===================================================================
--- Python/pythonrun.c  (révision 67870)
+++ Python/pythonrun.c  (copie de travail)
@@ -810,7 +810,12 @@
 #endif
        }
        else {
-               if (!(std = PyFile_FromFd(fd, "<stdout>", "w", -1, 
encoding,
+               int buffering;
+               if (1)
+                       buffering = 1; /* line */
+               else
+                       buffering = -1; /* default */
+               if (!(std = PyFile_FromFd(fd, "<stdout>", "w", 
buffering, encoding,
                                          errors, "\n", 0))) {
                        goto error;
                }

But "if (1)" have to be replaced "if -u option is used" :-) See 
unbuffered variable of Modules/main.c.
msg78127 - (view) Author: Fabio Zadrozny (fabioz) Date: 2008-12-20 21:51
Just as a note, Pydev needs the unbuffered output (or it cannot get it).
This has been brought up in the python-dev list:
http://mail.python.org/pipermail/python-dev/2008-December/084436.html

As a workaround for now I'm using:
sys.stdout._line_buffering = True, 

but that doesn't seem right as it's accessing an internal attribute.
msg78387 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-27 22:49
Here is a patch.
msg78393 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-28 01:23
pitrou's patch changes PyFile_FromFd() behaviour for a text file 
opened with buffering=0:
  /* As a convenience, when buffering == 0 on a text file, we
     open the underlying binary stream in unbuffered mode and
     wrap it with a text stream in line-buffered mode. */

Why changing PyFile_FromFd() and not io.open() directly?

Note: I prefer Py_UnbufferedStdoutFlag=1 instead of 
Py_UnbufferedStdoutFlag++ (for -u command line option).

Except the minor comments, I like the patch (and it has unit 
tests!) ;-)
msg78395 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-28 10:27
> pitrou's patch changes PyFile_FromFd() behaviour for a text file 
> opened with buffering=0:
>   /* As a convenience, when buffering == 0 on a text file, we
>      open the underlying binary stream in unbuffered mode and
>      wrap it with a text stream in line-buffered mode. */
> 
> Why changing PyFile_FromFd() and not io.open() directly?

I must admit I'm a bit lazy, and changing io.open() means changing a
fundamental public API, as Guido said on python-dev, so more discussion
and some parts of the patches delayed to 3.1. If someone else wants to
do it, please don't hesitate...

> Note: I prefer Py_UnbufferedStdoutFlag=1 instead of 
> Py_UnbufferedStdoutFlag++ (for -u command line option).

Well, I minimally changed the existing code.
msg78400 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-28 12:19
> > Why changing PyFile_FromFd() and not io.open() directly?
>
> I must admit I'm a bit lazy, and changing io.open() means changing 
> a fundamental public API, as Guido said on python-dev, so 
> more discussion and some parts of the patches delayed to 3.1.

You're right, and PyFile_FromFd() is also a fundamental "public" API. 
Since TextIOWrapper doesn't support real unbuffered buffer (only 
pseudo line buffer: unbuffered raw buffer and line buffering for 
TextIOWrapper), I prefer to change only stdout/stderr instead of 
PyFile_FromFd(). 

My new patch only changes initstdio() using pitrou's code.

Should we also change stdin?
msg78403 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-28 13:54
Le dimanche 28 décembre 2008 à 12:19 +0000, STINNER Victor a écrit :
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> > > Why changing PyFile_FromFd() and not io.open() directly?
> >
> > I must admit I'm a bit lazy, and changing io.open() means changing 
> > a fundamental public API, as Guido said on python-dev, so 
> > more discussion and some parts of the patches delayed to 3.1.
> 
> You're right, and PyFile_FromFd() is also a fundamental "public" API. 

Well, open() is fundamental as in part of the built-ins and used
pervasively. PyFile_FromFd(), on the other hand, is a relic of the 2.x C
file handling API. Let's see what others have to say about this.

> Should we also change stdin?

I don't know, but "python -h" only talks about stderr/stdout.
msg78408 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-12-28 15:11
It seems the "name" field of the TextIOWrapper object isn't set in
create_stdio() (the "char *name" parameter isn't used). Otherwise, the
patch looks good.
msg78417 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-28 16:04
>> Should we also change stdin?
> I don't know, but "python -h" only talks about stderr/stdout.

The manpage of Python2 is clear:

   -u     Force stdin, stdout and stderr to be totally unbuffered.

stdin is also unbuffered.

> It seems the "name" field of the TextIOWrapper object isn't 
> set in create_stdio()

It used only used for buffered output. Without the patch, 
sys.stdout.name == sys.stdout.buffer.name == '1' :-/

New patch:
 - use create_stdio() to create unbuffered sys.stdin
 - rename Py_UnbufferedStdoutFlag to Py_UnbufferedStdioFlag
 - replace "Py_UnbufferedStdioFlag++;" by "Py_UnbufferedStdioFlag = 
1;"
 - change create_stdio(): (...)

Note: there is no test for unbuffered input because I don't know how 
to test this (even by manual tests) :-p
msg78420 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2008-12-28 16:57
Attached: quick and dirty test to check if the standard input is 
buffered or not. My short test program works with python2.5 and py3k 
trunk without the -u command line option. So changing sys.stdin buffer 
is not really important.

About the wrong name, I opened a separated issue: #4762, 
PyFile_FromFd() doesn't set the file name.
msg78886 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-01-02 21:51
Instead of importing IO each time in create_stdio, maybe you should just
pass io.open to create_stdio.
msg78922 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-03 01:40
> Instead of importing IO each time in create_stdio, 
> maybe you should just pass io.open to create_stdio

create_stdio() uses io.open() but also io.TextIOWrapper. Since io 
module is already imported in initstdio(), I updated the patch to just 
pass the pointer to the module to create_stdio().
msg79444 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-08 21:37
If `PyObject_SetAttrString(raw, "_name", text)` fails, a reference to
raw is leaked.
Other than that, the patch looks good.
msg79449 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2009-01-08 23:18
Updated patch: clear raw on error
+	if (!Py_UnbufferedStdioFlag)
+		Py_XDECREF(raw);

Question: Should we use line_buffering in unbuffered mode?
msg79490 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-09 19:15
Committed in r68451. Thanks!
msg80190 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-19 16:27
Reopening, since sys.stdin is actually broken in unbuffered mode:

$ ./python -u
Python 3.1a0 (py3k:68756, Jan 19 2009, 01:17:26) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdin.read(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/py3k/__svn__/Lib/io.py", line 1739, in read
    eof = not self._read_chunk()
  File "/home/antoine/py3k/__svn__/Lib/io.py", line 1565, in _read_chunk
    input_chunk = self.buffer.read1(self._CHUNK_SIZE)
AttributeError: 'FileIO' object has no attribute 'read1'
>>> 

What I propose is that stdin be always opened in buffered mode (even
with -u), since I don't see how the behaviour can differ for a read-only
non-seekable stream.
msg80195 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-19 17:15
Here is a patch.
msg80539 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-25 21:25
> since I don't see how the behaviour can differ for a read-only
> non-seekable stream.

Unless I'm misunderstanding you (quite likely), I think one *can* get 
different results with buffered and unbuffered stdin.
For example, on my machine, if I create the following script:

#!/usr/bin/python -u
import sys
print sys.stdin.readline()

and name it test.py, I get the following result in an OS X Terminal 
running bash:

dickinsm$ ls python_source/trunk/Objects/ | (./test.py; ./test.py)
abstract.c

boolobject.c

Whereas if I remove the '-u' from the shebang line I just get:

dickinsm$ ls python_source/trunk/Objects/ | (./test.py; ./test.py)
abstract.c


I'm not 100% sure that I understand exactly what's going on here, but it's 
something like the following:  in the first (unbuffered) case, the 
stdin.readline call of the first ./test.py only reads one line from stdin, 
leaving the rest intact;  so the second ./test.py also gets to output a 
line.  In the second case some larger amount of stdin (1024 bytes?) is 
immediately slurped into the stdin buffer for the first Python process, so 
the second ./test.py doesn't get anything.
msg80540 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-25 21:33
[...]

I hadn't thought of such situations :-/

So the question is whether it is really useful to enforce unbuffered
stdin with the '-u' option (or your example is simply too borderline).
If so, the patch will have to be replaced with another one implementing
read1() in the FileIO class.
msg80541 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-25 21:36
Thinking about it, TextIOWrapper has its own input buffering (the
`decoded_chars` attribute), so your use case would probably not be
satisfied.

(and disabling TextIOWrapper's internal buffering would be a bad idea
since it would make it horribly slow)
msg80543 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-25 22:00
> So the question is whether it is really useful to enforce unbuffered
> stdin with the '-u' option (or your example is simply too borderline).

Hard to say.  It seems at least possible that there are Python users for 
whom stdin being unbuffered (with -u) matters, so if there's any 
reasonable way of avoiding changing this it should probably be considered.

Though I have to admit that I'm not one of those users (I don't think I've 
*ever* used the -u option outside of testing...).
msg80544 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-25 22:10
> Hard to say.  It seems at least possible that there are Python users for 
> whom stdin being unbuffered (with -u) matters, so if there's any 
> reasonable way of avoiding changing this it should probably be considered.

It's not about changing it, stdin has always been buffered in py3k. My
original commit actually attempted to change it, and it failed (I hadn't
noticed it at first due to mis-testing on my part). The new patch is
about putting it back in buffered mode even with '-u'.
msg80575 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2009-01-26 16:05
> It's not about changing it, stdin has always been buffered in py3k.

Sorry:  I should have been clearer.  It's the change from 2.x to 3.x that 
interests me.

So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not;  I'm 
wondering: was this an intentional design change?  Or was it just an 
accident/by-product of the rewritten io?

Anyway, the patch looks good to me.
msg80590 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-26 20:49
> So 'python3.0 -u' has buffered stdin, while 'python2.6 -u' does not;
> I'm wondering: was this an intentional design change?  Or was it just
> an accident/by-product of the rewritten io?

I'm not sure (I didn't write the new io in the first place) but I'd say
it was simply overlooked. Otherwise 'python3.0 -u' would have had at
least unbuffered stdout/stderr, which it didn't have.
msg80597 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-26 22:09
Committed and applied a small fix to the test so that it passes in debug
mode (r68977, r68981, r68982). Thanks!
msg196224 - (view) Author: Joe Borg (Joe.Borg) Date: 2013-08-26 16:45
Can I confirm this is still in the trunk?  I have 3.3.2 and am suffering from the fact that `-u` isn't setting stdin to unbuffered.  I'm have to run a flush every command, which is awful.
History
Date User Action Args
2013-08-26 16:45:16Joe.Borgsetnosy: + georg.brandl, Joe.Borg

messages: + msg196224
versions: + Python 3.2, Python 3.3
2009-01-26 22:09:35pitrousetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg80597
2009-01-26 21:19:24pitrousetassignee: pitrou
resolution: accepted
2009-01-26 20:49:18pitrousetmessages: + msg80590
2009-01-26 16:05:07mark.dickinsonsetmessages: + msg80575
2009-01-25 22:10:09pitrousetmessages: + msg80544
2009-01-25 22:00:38mark.dickinsonsetmessages: + msg80543
2009-01-25 21:36:56pitrousetmessages: + msg80541
2009-01-25 21:33:39pitrousetmessages: + msg80540
2009-01-25 21:25:19mark.dickinsonsetnosy: + mark.dickinson
messages: + msg80539
2009-01-20 22:10:05pitrousetfiles: - unbufferedstdout-5.patch
2009-01-20 22:10:02pitrousetfiles: - unbufferedstdout-4.patch
2009-01-20 22:09:34pitrousetfiles: - test_stdin.py
2009-01-20 22:09:30pitrousetfiles: - unbufferedstdout.patch
2009-01-20 22:09:17pitrousetpriority: high -> release blocker
2009-01-19 17:15:58pitrousetfiles: + unbuffered-stdin.patch
messages: + msg80195
2009-01-19 16:27:02pitrousetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg80190
stage: commit review -> resolved
2009-01-09 19:15:24pitrousetstatus: open -> closed
resolution: accepted -> fixed
messages: + msg79490
2009-01-09 18:45:07pitrousetstage: patch review -> commit review
resolution: accepted
versions: + Python 3.1
2009-01-08 23:18:09vstinnersetfiles: + unbufferedstdout-5.patch
messages: + msg79449
2009-01-08 21:37:05pitrousetmessages: + msg79444
2009-01-03 01:40:16vstinnersetfiles: - unbufferedstdout-3.patch
2009-01-03 01:40:11vstinnersetfiles: + unbufferedstdout-4.patch
messages: + msg78922
2009-01-02 23:42:19vstinnersetfiles: - unbufferedstdout-2.patch
2009-01-02 21:51:29benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg78886
2008-12-28 16:57:48vstinnersetfiles: + test_stdin.py
messages: + msg78420
2008-12-28 16:04:52vstinnersetfiles: + unbufferedstdout-3.patch
messages: + msg78417
2008-12-28 15:11:09pitrousetmessages: + msg78408
2008-12-28 13:54:03pitrousetmessages: + msg78403
2008-12-28 12:19:18vstinnersetfiles: + unbufferedstdout-2.patch
messages: + msg78400
2008-12-28 10:27:50pitrousetmessages: + msg78395
2008-12-28 01:23:50vstinnersetmessages: + msg78393
2008-12-27 22:49:41pitrousetfiles: + unbufferedstdout.patch
nosy: + pitrou
messages: + msg78387
priority: high
keywords: + patch
type: behavior
stage: patch review
2008-12-20 21:51:59fabiozsetnosy: + fabioz
messages: + msg78127
2008-12-20 13:47:22vstinnercreate