Title: python --help: -u is misdocumented as binary mode
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.5
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: berker.peksag, eryksun,, haypo, ncoghlan, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-11-09 10:26 by arigo, last changed 2017-05-24 21:38 by arigo.

File name Uploaded Description Edit
issue28647.patch, 2016-11-12 18:36 review
Pull Requests
URL Status Linked Edit
PR 1655 open berker.peksag, 2017-05-18 19:49
PR 1796 closed matrixise, 2017-05-24 20:57
PR 1797 closed matrixise, 2017-05-24 21:17
Messages (10)
msg280390 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2016-11-09 10:26
``python3.5 --help`` gives this information:

    -u : unbuffered binary stdout and stderr, stdin always buffered

However, stdout and stderr are actually always opened in text mode, and print() always expects a string and never a bytes object.  This usage of "binary" in the --help is in contradiction with the usage of "binary" in the description of files (e.g. ``help(open)``).
msg280391 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-11-09 10:30
Right. Moreover, "unbuffered" is wrong. It's line buffered: sys.stdout.buffer is directly a io.FileIO object, but TextIOWrapper only calls buffer.write() when the message contains a newline character. It's not (currently) possible to have a fully unbuffered stdout.
msg280392 - (view) Author: Eryk Sun (eryksun) * Date: 2016-11-09 11:04
> It's not (currently) possible to have a fully unbuffered stdout.

Why doesn't create_stdio also pass `write_through = Py_True` when Py_UnbufferedStdioFlag is set? This would immediately pass writes through to the FileIO object, even without containing a newline (i.e. it sets text_needflush in _io_TextIOWrapper_write_impl).
msg280393 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-11-09 12:47
I don't recall, I would have to search in old issues for the rationale. But
the explanation is probably performance, reduce the number of syscalls.
msg280394 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-09 13:18
The write_through argument was added only in 3.3. I think this is a good idea to pass `write_through = Py_True` when Py_UnbufferedStdioFlag is set. Using -u means that performance is not important (that is why this is not default behavior).
msg280405 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2016-11-09 14:58
Would it make sense to have two modes: line buferred and unbuffered,
as the C function setvbuf() for stdout?
msg280665 - (view) Author: Gareth Rees ( * Date: 2016-11-12 18:21
The output of "python3.5 --help" says:

    -u : unbuffered binary stdout and stderr, stdin always buffered;
         also PYTHONUNBUFFERED=x
         see man page for details on internal buffering relating to '-u'

If you look at the man page as instructed then you'll see a clearer

    -u   Force  the  binary  I/O  layers  of  stdout  and  stderr  to  be
         unbuffered.  stdin is always buffered.  The text I/O layer  will
         still be line-buffered.

For example, if you try this:

    python3.5 -uc 'import sys,time;w=sys.stdout.buffer.write;w(b"a");time.sleep(1);w(b"b");'

then you'll see that the binary output is indeed unbuffered as

The output of --help is trying to abbreviate this explanation, but I
think it's abbreviated too much. The explanation from the man page
seems clear to me, and is only a little longer, so I suggest changing
the --help output to match the man page.
msg280666 - (view) Author: Gareth Rees ( * Date: 2016-11-12 18:36
Here's a patch that copies the text for the -u option from the man page to the --help output.
msg289649 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2017-03-15 08:54
I just ran into this discrepancy working on the test cases for PEP 538 - +1 for Gareth's suggested approach of just aligning the `--help` output with the man page.
msg293961 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-19 16:05
Issue30404 makes stdout and stderr truly unbuffered when run with -u.
Date User Action Args
2017-05-24 21:38:18arigosetnosy: - arigo
2017-05-24 21:17:35matrixisesetpull_requests: + pull_request1880
2017-05-24 20:57:02matrixisesetpull_requests: + pull_request1879
2017-05-19 16:05:37serhiy.storchakasetmessages: + msg293961
2017-05-18 19:49:11berker.peksagsetpull_requests: + pull_request1749
2017-03-15 08:54:59ncoghlansetnosy: + ncoghlan
messages: + msg289649
2017-01-07 07:55:23berker.peksagsetnosy: + berker.peksag
stage: patch review
type: behavior

versions: + Python 3.6, Python 3.7
2016-11-12 18:36:51gdr@garethrees.orgsetfiles: + issue28647.patch
keywords: + patch
messages: + msg280666
2016-11-12 18:21:35gdr@garethrees.orgsetnosy: +
messages: + msg280665
2016-11-09 14:58:38hayposetmessages: + msg280405
2016-11-09 13:18:28serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg280394
2016-11-09 12:47:02hayposetmessages: + msg280393
2016-11-09 11:04:55eryksunsetnosy: + eryksun
messages: + msg280392
2016-11-09 10:30:40hayposetnosy: + haypo
messages: + msg280391
2016-11-09 10:26:05arigocreate