classification
Title: Improve documentation of stdout/stderr buffering in Python 3.x
Type: behavior Stage: resolved
Components: Documentation, IO Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: benjamin.peterson, docs@python, gjb1002, pitrou, pjenvey, python-dev, stutzbach
Priority: normal Keywords:

Created on 2011-12-13 20:28 by gjb1002, last changed 2011-12-21 20:50 by gjb1002. This issue is now closed.

Messages (13)
msg149408 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-13 20:28
The default buffering of standard output and standard error has changed in Python 3.x with respect to Python 2.x, and I have been unable to find decent documentation of either the current behaviour, or the change.

(See also http://groups.google.com/group/comp.lang.python/browse_thread/thread/43476d4682059f53#)

Part 1 - the change
-------------------
From rude experiment it seems that:
a) In Python 2.x, standard error was always unbuffered while standard
output was buffered by default.  In python3, both are buffered. In
both cases, "buffered" means line-buffered when writing to the console
and not line-buffered when redirected to files.
b) In Python 2.x, the "-u" flag meant everything was totally
unbuffered. In Python 3.x, it means that both stdout and stderr are
line-buffered also when redirected to files. 

One important consequence of (a) is, if stderr is redirected
to a file, your program throws an exception and is then subsequently terminated with SIGTERM, you will not see the exception. This will not be expected for someone used to the Python 2.x behaviour.

"What's New in Python 3.0" has this to say about the change (in the section marked "Changes Already Present In Python 2.6"

"# PEP 3116: New I/O Library. The io module is now the standard way of doing file I/O, and the initial values of sys.stdin, sys.stdout and sys.stderr are now instances of io.TextIOBase. [...]"

This seems wrong in that, while the io module was present in Python 2.6, the change noted to sys.stdin, sys.stdout and sys.stderr was not.
Also, it is far from obvious from this note that any externally observable behaviour has changed.

I suggest changing this to 
a) note the buffering changes listed above
b) note the change in meaning of the -u flag
c) Move this to its own section which is not part of changes to Python 2.6 (it's OK to keep the note about the new io module there)

Part 2 - the behaviour
----------------------
a) The documentation for "sys.stdout" and "sys.stderr" does not say anything about their default buffering properties in various situations, nor how this can modified by setting the "-u" flag.

b) The documentation for "-u" is misleading:
"Force the binary layer of the stdin, stdout and stderr streams (which
is available as their buffer attribute) to be unbuffered. The text I/O
layer will still be line-buffered." 

The "still" in the last sentence is only relevant when stdout/stderr are writing to the console. If they are redirected to file, "-u" *modifies the behaviour such that* the text I/O layer will be line-buffered.
msg149415 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-14 02:07
We could force sys.stderr to be always line-buffered if that's better than the current settings.
msg149419 - (view) Author: Philip Jenvey (pjenvey) * (Python committer) Date: 2011-12-14 03:22
I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately

Was this an oversight in migrating stdin/out/err to the new io module?
msg149446 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-14 13:07
> I'm surprised to hear that stderr is line buffered by default.
> Historically stderr is never buffered (at least on POSIX) and for good
> reason: errors should be seen immediately
> 
> Was this an oversight in migrating stdin/out/err to the new io module?

Until recently it wasn't possible to have an unbuffered text stream with
the new io stack. Now it's possible in write-only mode, although the
open() builtin hasn't been updated for it.
msg149464 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-14 18:38
I would certainly be in favour of restoring the python 2.x behaviour, at least where standard error is concerned. When writing Python programs, it's important to see exceptions immediately, and not lose them entirely in some circumstances.

I reported this as a documentation bug because I got the impression it was deliberate, mostly from reading http://bugs.python.org/issue4705 and the comp.lang.python thread in the description, but I personally think the Python 2.x behaviour was preferable.
msg149465 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-14 18:41
Ooops, seems like I just ran into a bug in the bug tracker too, it seems to have backed out other people's changes. Restoring them...
msg149477 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-14 21:15
(I've opened issue13601 for the possible behaviour change)
msg149563 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-12-15 15:26
New changeset 313fa7ea6c79 by Antoine Pitrou in branch '3.2':
Issue #13597: Improve documentation of standard streams.
http://hg.python.org/cpython/rev/313fa7ea6c79

New changeset 7343730185a3 by Antoine Pitrou in branch 'default':
Issue #13597: Improve documentation of standard streams.
http://hg.python.org/cpython/rev/7343730185a3
msg149564 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-15 15:27
Should be ok now.
msg149888 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-19 20:05
The changes are good as far as they go, but they only affect the documentation of sys.stderr and sys.stdout. 

I also suggested changes to the documentation of the "-u" flag, and to "What's New in Python 3.0", can someone look at that also?
msg149978 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2011-12-21 10:25
New changeset 19df72a77b39 by Antoine Pitrou in branch '3.2':
Issue #13597: Fix the documentation of the "-u" command-line option, and wording of "What's new in Python 3.0" about standard streams.
http://hg.python.org/cpython/rev/19df72a77b39

New changeset 1ab124a6f171 by Antoine Pitrou in branch 'default':
Issue #13597: Fix the documentation of the "-u" command-line option, and wording of "What's new in Python 3.0" about standard streams.
http://hg.python.org/cpython/rev/1ab124a6f171
msg149979 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-21 10:25
> I also suggested changes to the documentation of the "-u" flag, and to 
> "What's New in Python 3.0", can someone look at that also?

Sorry for having overlooked that. This should now be fixed as well.
msg150043 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-21 20:50
Thanks.

I'm not sure what you've written about the "-u" flag is correct though currently. From experimenting I believe it changes buffering of stdout and stderr to line-buffering also when directed to file, i.e. it does affect the behaviour of the text-layer.

Some other changes might be needed also, but perhaps they should wait until we know whether issue13601 will be accepted.
History
Date User Action Args
2011-12-21 20:50:52gjb1002setmessages: + msg150043
2011-12-21 10:25:59pitrousetmessages: + msg149979
2011-12-21 10:25:31python-devsetmessages: + msg149978
2011-12-19 20:05:07gjb1002setmessages: + msg149888
2011-12-15 15:27:54pitrousetstatus: open -> closed
resolution: fixed
messages: + msg149564

stage: resolved
2011-12-15 15:26:31python-devsetnosy: + python-dev
messages: + msg149563
2011-12-14 21:15:00pitrousetmessages: + msg149477
2011-12-14 18:41:02gjb1002setversions: - Python 3.1, Python 3.4
nosy: + pitrou, pjenvey, benjamin.peterson, stutzbach

messages: + msg149465

components: + IO
2011-12-14 18:38:01gjb1002setversions: + Python 3.1, Python 3.4
nosy: - pitrou, pjenvey, benjamin.peterson, stutzbach

messages: + msg149464

components: - IO
2011-12-14 13:07:42pitrousetmessages: + msg149446
2011-12-14 03:22:02pjenveysetnosy: + pjenvey
messages: + msg149419
2011-12-14 02:07:47pitrousetversions: - Python 3.1, Python 3.4
nosy: + pitrou, benjamin.peterson, stutzbach

messages: + msg149415

components: + IO
2011-12-13 20:28:02gjb1002create