classification
Title: Make stdout and stderr truly unbuffered when using -u option
Type: enhancement Stage: resolved
Components: Interpreter Core, IO Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, berker.peksag, pitrou, serhiy.storchaka, stutzbach, vstinner
Priority: normal Keywords:

Created on 2017-05-19 15:52 by serhiy.storchaka, last changed 2017-10-13 12:16 by berker.peksag. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1667 merged serhiy.storchaka, 2017-05-19 16:02
PR 1655 berker.peksag, 2017-05-24 22:13
PR 3961 merged berker.peksag, 2017-10-12 05:58
Messages (9)
msg293960 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-19 15:52
In Python 2 when run the interpreter with the -u option the stdout and stderr streams are unbuffered. In Python 3 they become just line-buffered. This is because initially there was no way to create unbuffered text streams. But since Python 3.3 TextIOWrapper supports unbuffered output binary stream and accepts the write_through argument which switch off its own buffering.

Proposed patch makes the stdout and stderr streams truly unbuffered when run with the -u option.
msg296315 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-19 11:46
Oh, I like https://github.com/python/cpython/pull/1667/ 

"If write_through is True, calls to write() are guaranteed not to be buffered: any data written on the TextIOWrapper object is immediately handled to its underlying binary buffer."

I didn't know write_through. It seems like it was introduced in Python 3.7: bpo-30526, commit 3c2817b6884a5fcf792197203f3c26b157210607.
msg296316 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-19 11:48
While I hope that users of the -u options expect the slow-down, would it be possible to benchmark it?

For example, try to write setup.py content character by character to stdout using -u or not, into a TTY, into a pipe and/or a file. I'm curious if the line buffering vs really unbuffered has a significant impact on performances.
msg296344 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-19 14:21
Writing separate lines:

$ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
200 loops, best of 5: 1.07 msec per loop

$ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = f.readlines()' 'sys.stderr.writelines(s)' 2>/dev/null
Unpatched:  50 loops, best of 5: 5.89 msec per loop
Patched:    100 loops, best of 5: 3.32 msec per loop

Writing separate characters:

$ ./python -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
10 loops, best of 5: 30 msec per loop

$ ./python -u -m timeit -s 'import sys' -s 'with open("setup.py") as f: s = list(f.read())' 'sys.stderr.writelines(s)' 2>/dev/null
Unpatched:  5 loops, best of 5: 49.2 msec per loop
Patched:    2 loops, best of 5: 137 msec per loop
msg296345 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-19 14:37
Hum. It has an huge impact on performances. Would it make sense to have two command line options to choose between unbuffered and line buffered?

The C setvbuf() function uses these constants:

              _IONBF unbuffered
              _IOLBF line buffered
              _IOFBF fully buffered
msg296352 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-19 16:08
Note that if output by lines, the patch speeds up the output! Actually the output is fast enough with buffering and without. The only slowdown is exposed when output by characters, but this is uncommon case. And if you output by characters (in case of drawing a progressbar or like), then perhaps you want the characters been displayed immediately, without buffering.

See also issue13601.
msg303716 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-04 17:24
If there is a need in making redirected stdout line-buffered, a new option can be added in separate issue.
msg303717 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-10-04 17:25
New changeset 77732be801c18013cfbc86e27fcc50194ca22c8e by Serhiy Storchaka in branch 'master':
bpo-30404: The -u option now makes the stdout and stderr streams totally unbuffered. (#1667)
https://github.com/python/cpython/commit/77732be801c18013cfbc86e27fcc50194ca22c8e
msg304332 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-10-13 12:16
New changeset 7f580970836b0f6bc9c5db868d95bea81a3e1558 by Berker Peksag in branch 'master':
bpo-28647: Update -u documentation after bpo-30404 (GH-3961)
https://github.com/python/cpython/commit/7f580970836b0f6bc9c5db868d95bea81a3e1558
History
Date User Action Args
2017-10-13 12:16:37berker.peksagsetnosy: + berker.peksag
messages: + msg304332
2017-10-12 05:58:58berker.peksagsetpull_requests: + pull_request3939
2017-10-04 17:26:10serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-10-04 17:25:42serhiy.storchakasetmessages: + msg303717
2017-10-04 17:24:42serhiy.storchakasetmessages: + msg303716
2017-06-19 16:08:15serhiy.storchakasetmessages: + msg296352
2017-06-19 14:37:03vstinnersetmessages: + msg296345
2017-06-19 14:21:07serhiy.storchakasetmessages: + msg296344
2017-06-19 11:48:59vstinnersetmessages: + msg296316
2017-06-19 11:46:51vstinnersetnosy: + vstinner
messages: + msg296315
2017-05-24 22:13:46berker.peksagsetpull_requests: + pull_request1881
2017-05-19 21:45:35terry.reedysettitle: Make stdout and stderr truly unbuffered when run with the -u option -> Make stdout and stderr truly unbuffered when using -u option
2017-05-19 16:02:43serhiy.storchakasetpull_requests: + pull_request1762
2017-05-19 15:52:05serhiy.storchakacreate