The default buffering of standard output and standard error has changed in Python 3.x with respect to Python 2.x, and I have been unable to find decent documentation of either the current behaviour, or the change.

(See also http://groups.google.com/group/comp.lang.python/browse_thread/thread/43476d4682059f53#)

Part 1 - the change
-------------------
From rude experiment it seems that:
a) In Python 2.x, standard error was always unbuffered while standard
output was buffered by default.  In python3, both are buffered. In
both cases, "buffered" means line-buffered when writing to the console
and not line-buffered when redirected to files.
b) In Python 2.x, the "-u" flag meant everything was totally
unbuffered. In Python 3.x, it means that both stdout and stderr are
line-buffered also when redirected to files. 

One important consequence of (a) is, if stderr is redirected
to a file, your program throws an exception and is then subsequently terminated with SIGTERM, you will not see the exception. This will not be expected for someone used to the Python 2.x behaviour.

"What's New in Python 3.0" has this to say about the change (in the section marked "Changes Already Present In Python 2.6"

"# PEP 3116: New I/O Library. The io module is now the standard way of doing file I/O, and the initial values of sys.stdin, sys.stdout and sys.stderr are now instances of io.TextIOBase. [...]"

This seems wrong in that, while the io module was present in Python 2.6, the change noted to sys.stdin, sys.stdout and sys.stderr was not.
Also, it is far from obvious from this note that any externally observable behaviour has changed.

I suggest changing this to 
a) note the buffering changes listed above
b) note the change in meaning of the -u flag
c) Move this to its own section which is not part of changes to Python 2.6 (it's OK to keep the note about the new io module there)

Part 2 - the behaviour
----------------------
a) The documentation for "sys.stdout" and "sys.stderr" does not say anything about their default buffering properties in various situations, nor how this can modified by setting the "-u" flag.

b) The documentation for "-u" is misleading:
"Force the binary layer of the stdin, stdout and stderr streams (which
is available as their buffer attribute) to be unbuffered. The text I/O
layer will still be line-buffered." 

The "still" in the last sentence is only relevant when stdout/stderr are writing to the console. If they are redirected to file, "-u" *modifies the behaviour such that* the text I/O layer will be line-buffered.

We could force sys.stderr to be always line-buffered if that's better than the current settings.

I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately

Was this an oversight in migrating stdin/out/err to the new io module?

> I'm surprised to hear that stderr is line buffered by default.
> Historically stderr is never buffered (at least on POSIX) and for good
> reason: errors should be seen immediately
> 
> Was this an oversight in migrating stdin/out/err to the new io module?

Until recently it wasn't possible to have an unbuffered text stream with
the new io stack. Now it's possible in write-only mode, although the
open() builtin hasn't been updated for it.

I would certainly be in favour of restoring the python 2.x behaviour, at least where standard error is concerned. When writing Python programs, it's important to see exceptions immediately, and not lose them entirely in some circumstances.

I reported this as a documentation bug because I got the impression it was deliberate, mostly from reading http://bugs.python.org/issue4705 and the comp.lang.python thread in the description, but I personally think the Python 2.x behaviour was preferable.

Ooops, seems like I just ran into a bug in the bug tracker too, it seems to have backed out other people's changes. Restoring them...

(I've opened issue13601 for the possible behaviour change)

New changeset 313fa7ea6c79 by Antoine Pitrou in branch '3.2':
Issue #13597: Improve documentation of standard streams.
http://hg.python.org/cpython/rev/313fa7ea6c79

New changeset 7343730185a3 by Antoine Pitrou in branch 'default':
Issue #13597: Improve documentation of standard streams.
http://hg.python.org/cpython/rev/7343730185a3

Should be ok now.

The changes are good as far as they go, but they only affect the documentation of sys.stderr and sys.stdout. 

I also suggested changes to the documentation of the "-u" flag, and to "What's New in Python 3.0", can someone look at that also?

New changeset 19df72a77b39 by Antoine Pitrou in branch '3.2':
Issue #13597: Fix the documentation of the "-u" command-line option, and wording of "What's new in Python 3.0" about standard streams.
http://hg.python.org/cpython/rev/19df72a77b39

New changeset 1ab124a6f171 by Antoine Pitrou in branch 'default':
Issue #13597: Fix the documentation of the "-u" command-line option, and wording of "What's new in Python 3.0" about standard streams.
http://hg.python.org/cpython/rev/1ab124a6f171

> I also suggested changes to the documentation of the "-u" flag, and to 
> "What's New in Python 3.0", can someone look at that also?

Sorry for having overlooked that. This should now be fixed as well.

Thanks.

I'm not sure what you've written about the "-u" flag is correct though currently. From experimenting I believe it changes buffering of stdout and stderr to line-buffering also when directed to file, i.e. it does affect the behaviour of the text-layer.

Some other changes might be needed also, but perhaps they should wait until we know whether issue13601 will be accepted.