This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: sys.stderr should be line-buffered when stderr is not a TTY
Type: behavior Stage: resolved
Components: Interpreter Core, IO, Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, gjb1002, gvanrossum, hauntsaninja, jendrik, miss-islington, ncoghlan, pitrou, pjenvey, serhiy.storchaka, stutzbach, torsten, vstinner
Priority: normal Keywords: patch

Created on 2011-12-14 13:11 by pitrou, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 17646 merged jendrik, 2019-12-17 22:12
PR 20168 merged hauntsaninja, 2020-05-18 04:52
Messages (21)
msg149447 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-14 13:11
In issue13597, Philip Jenvey points out:

“I'm surprised to hear that stderr is line buffered by default. Historically stderr is never buffered (at least on POSIX) and for good reason: errors should be seen immediately”

Recent changes to the IO stack should allow stderr to be opened in fully unbuffered mode (and open(..., 'w', buffering=0) can be allowed too). Or at least it could be always line-buffered, even when redirected to a file.
msg149454 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2011-12-14 15:29
I *thought* I mimicked what C stdio did ~20 years ago...  I'd be happy to follow what it does today if it changed or if I made a mistake.

That said, IMO:

Line-buffering should be good enough since in practice errors messages are always terminated by a newline.

I'm hesitant to make it line-buffered by default when directed to a file, since this could significantly slow down a program that for some reason produces super-voluminous output (e.g. when running a program with heavy debug logging turned on).

Maybe we need better command-line control to override the defaults?  Are there precedents e.g. in Bash flags?
msg149560 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-15 14:27
> Line-buffering should be good enough since in practice errors messages
> are always terminated by a newline.

What I think too.

> I'm hesitant to make it line-buffered by default when directed to a
> file, since this could significantly slow down a program that for some
> reason produces super-voluminous output (e.g. when running a program
> with heavy debug logging turned on).

The slow-down is impressive in relative terms (6x) but the timings are
still small in absolute value:

$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=4096)" "f.write('log message\n')"
10000000 loops, best of 3: 0.156 usec per loop
$ ./python -m timeit -s "f=open('/dev/null', 'a', buffering=1)" "f.write('log message\n')"
1000000 loops, best of 3: 0.961 usec per loop
msg149561 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-12-15 14:30
Oops, I forgot the last two questions:

> Maybe we need better command-line control to override the defaults?

We already have -u to switch all stdio to unbuffered. This issue proposes to make stderr line-buffered/unbuffered by default, since it's less surprising than fully buffered.

> Are there precedents e.g. in Bash flags?

Well, `man bash` doesn't appear to say anything about stdio buffering.
msg149887 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-19 19:54
> I'm hesitant to make it line-buffered by default when directed to a 
> file, since this could significantly slow down a program that for some
> reason produces super-voluminous output (e.g. when running a program
> with heavy debug logging turned on).

Is that really the purpose of standard error though? Heavy debug output, in my experience, is usually sent to standard output or to another file.

Also, did anyone ever complain about this as a problem, given it is the default behaviour of Python 2?

In my view the requirements of seeing errors when they happen, and guaranteeing that they will always be seen no matter what happens afterwards, should weigh more heavily than this.
msg149889 - (view) Author: Geoffrey Bache (gjb1002) Date: 2011-12-19 20:09
I think we all agree line-buffering is sufficient, so I change the title.
msg254027 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-11-03 22:20
As I read this, there was agreement that the status quo is sufficient.  That would imply that this should be closed.  Correct?
msg254028 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-11-03 22:24
A month after this discussion, the flush keyword was added to print, which cover partial lines sent to either stdout or stderr via print.  https://bugs.python.org/issue13761
msg261852 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-03-16 12:57
This question came up today in the context of the final line of a traceback output potentially being missing from stderr if the closing flush of the standard streams is missed for any reason.

That's not going to be a common scenario (as far as I know it was an entirely hypothetical discussion), but the last line of a traceback is the one with the actual error message, so it's likely to be annoyingly cryptic if it does happen.
msg261853 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2016-03-16 13:12
Changing the target version and summarising my understanding of the status quo:

"python3": sys.stderr is line buffered at both the TextIOWrapper layer and may be fully buffered at the binary BufferedWriter layer if the output is redirected to a file
"python3 -u": the BufferedWriter layer is omitted entirely, leaving only the line buffering at the TextIOWrapper layer

Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

That means either line buffering or no buffering is considered acceptable, but full buffering is not.

So, at the very least, it seems to me that the way we configure stderr should be the same regardless of whether or not "-u" is used: omit the BufferedWriter layer.

Given that POSIX allows compliant implementations to use line-buffering on stderr, the "missing trailing newline and no implicit flush() is ever triggered" scenario is probably obscure enough not to worry about.
msg261916 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-03-17 16:32
I changed the title, since sys.stderr is already line-buffered when stderr is a TTY.
msg296303 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-19 03:31
See also issue28647 and issue30404.
msg296655 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-22 21:59
Amusingly, I didn't realize I had already opened this issue when one of our users hit it recently which led me to add TextIOWrapper.reconfigure(): https://bugs.python.org/issue30526

Still, I think it would be a good idea to do this as well (switch sys.stderr to line-buffered unconditionally), especially now that Nick found a POSIX reference that states C stderr should never be fully buffered.
msg304064 - (view) Author: Torsten Landschoff (torsten) * Date: 2017-10-10 18:05
> Looking at http://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html (which also covers stdout and stderr), it specifically says about stderr: "When opened, the standard error stream is not fully buffered;".

I was of the impression that this is defined in ISO C already. Unfortunately, I only have ISO C 99 at hand, but this clearly states in section 7.19.3 (Files), enumeration item 7:

> As initially opened,  the  standard  error  stream  is  not  fully  buffered;  the  standard  input  and  standard output streams are fully buffered if and only if the stream can be determined not to refer
to an interactive device.

I am quite sure this is just as as it was in the original ANSI C standard.
msg358593 - (view) Author: Jendrik Seipp (jendrik) * Date: 2019-12-17 22:17
I took the liberty of increasing the target version. It would be great if someone could review my patch for this issue at https://github.com/python/cpython/pull/17646 .
msg359171 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-01-01 22:21
New changeset 5b9077134cd0535f21905d5f5195847526cac99c by Antoine Pitrou (Jendrik Seipp) in branch 'master':
bpo-13601: always use line-buffering for sys.stderr (GH-17646)
https://github.com/python/cpython/commit/5b9077134cd0535f21905d5f5195847526cac99c
msg359172 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2020-01-01 22:22
Jendrik, thank you for fixing this!
msg359527 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-07 17:35
So it just took 9 years to fix this old bug :-)
msg369004 - (view) Author: Shantanu (hauntsaninja) * Date: 2020-05-16 03:11
I'm wondering if this should be mentioned in Python 3.9's What's New, potentially at https://docs.python.org/3.9/whatsnew/3.9.html#sys ?

This change broke one of mypy's tests on 3.9 and it was a little tricky to find what had changed.
msg369159 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-18 01:41
Can you submit a PR and CC me?
msg369177 - (view) Author: miss-islington (miss-islington) Date: 2020-05-18 05:08
New changeset d17f3d8315a3a775ab0807fc80acf92b1bd682f8 by Shantanu in branch 'master':
bpo-13601: Mention stderr's line buffering in What's New (GH-20168)
https://github.com/python/cpython/commit/d17f3d8315a3a775ab0807fc80acf92b1bd682f8
History
Date User Action Args
2022-04-11 14:57:24adminsetgithub: 57810
2020-05-18 05:08:00miss-islingtonsetnosy: + miss-islington
messages: + msg369177
2020-05-18 04:52:38hauntsaninjasetpull_requests: + pull_request19467
2020-05-18 01:41:38gvanrossumsetmessages: + msg369159
2020-05-16 03:11:55hauntsaninjasetnosy: + hauntsaninja
messages: + msg369004
2020-01-07 17:35:25vstinnersetmessages: + msg359527
2020-01-01 22:22:26pitrousetstatus: open -> closed
resolution: fixed
messages: + msg359172

stage: patch review -> resolved
2020-01-01 22:21:49pitrousetmessages: + msg359171
2019-12-17 22:17:59jendriksetnosy: + jendrik

messages: + msg358593
versions: + Python 3.9, - Python 3.7
2019-12-17 22:12:05jendriksetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request17114
2017-10-10 18:05:57torstensetnosy: + torsten
messages: + msg304064
2017-06-22 21:59:53pitrousetmessages: + msg296655
versions: + Python 3.7, - Python 3.6
2017-06-19 03:31:33serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg296303
2017-06-18 18:07:28terry.reedysetnosy: - terry.reedy
2016-03-17 16:32:47vstinnersetmessages: + msg261916
2016-03-17 16:32:13vstinnersettitle: sys.stderr should always be line-buffered -> sys.stderr should be line-buffered when stderr is not a TTY
2016-03-16 22:54:37vstinnersetnosy: + vstinner
2016-03-16 13:12:55ncoghlansetmessages: + msg261853
versions: + Python 3.6, - Python 3.2, Python 3.3
2016-03-16 12:57:02ncoghlansetnosy: + ncoghlan
messages: + msg261852
2015-11-03 22:24:23terry.reedysetmessages: + msg254028
2015-11-03 22:20:11terry.reedysetnosy: + terry.reedy
messages: + msg254027
2011-12-19 20:09:25gjb1002setmessages: + msg149889
title: sys.stderr should be unbuffered (or always line-buffered) -> sys.stderr should always be line-buffered
2011-12-19 19:54:43gjb1002setmessages: + msg149887
2011-12-15 14:30:31pitrousetmessages: + msg149561
2011-12-15 14:27:01pitrousetmessages: + msg149560
2011-12-15 08:05:03gjb1002setnosy: + gjb1002
2011-12-14 15:29:54gvanrossumsetmessages: + msg149454
2011-12-14 13:11:31pitroucreate