classification
Title: subprocess docs lack info how to use output result
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ronaldoussoren Nosy List: amaury.forgeotdarc, asvetlov, docs@python, ezio.melotti, python-dev, r.david.murray, ronaldoussoren, serhiy.storchaka, techtonik, tshepang, vstinner
Priority: normal Keywords:

Created on 2013-04-28 14:36 by techtonik, last changed 2013-07-07 07:28 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
issue17860.txt ronaldoussoren, 2013-06-12 14:57 review
Messages (15)
msg187982 - (view) Author: anatoly techtonik (techtonik) Date: 2013-04-28 14:36
http://docs.python.org/3/library/subprocess.html

A common confusion is that result from subprocess calls in Python 3 is bytes, not string, and needs to be converted.

The problem is complicated because you need to know the encoding of input/output streams. This should be documented at least.

http://stackoverflow.com/questions/606191/convert-byte-array-to-python-string
msg187996 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2013-04-28 16:14
Actually stdin/stdout/stderr are string streams if universal_newline is True
msg188016 - (view) Author: anatoly techtonik (techtonik) Date: 2013-04-28 19:19
>
>
> Actually stdin/stdout/stderr are string streams if universal_newline is
> True
>

I believe that makes the issue even worse. =)
msg188025 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-04-28 21:21
Anatoly, do you have a specific suggestion for improved wording?  This *is* documented in the subprocess documentation, in several appropriate places, including the fact that the appropriate encoding to use may need to be determined at the application level.
msg191032 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-12 14:57
The attached patch explicitly mentions that stdin/stdout/stderr are opened as binary streams when universal_newlines is False. 

I'm not convinced that adding this text will solve the confusion, subprocess has fairly complex documentation due to the rich API and it is easy to mis information when you don't read carefully. That said, the documentation for check_output (which is likely the most common way to run a command and fetch its output) is already clear about the types of the returned data, both in prose and example code.
msg191045 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-12 19:48
The only thing that can save the docs is pictures. Or tables.

          +---------+---------------------------+
          |         | universal_newlines = True |
   +------+---------+---------------------------+
   |Py.2  |   str   |         str               |
   +------+         |                           |
   |Py.3  |  bytes  |         str               |
   +------+---------+---------------------------+
msg191058 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-06-13 06:02
I don't agree that adding a table/picture is the only thing that can be done, or even that it is a good idea.

IMHO the subprocess documentation is clear enough about the distinction between bytes and string, especially in the section about convenience functions that is right at the top of the page.
msg191059 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-06-13 06:20
Also in the Python 3 docs we don't compare the current behavior with Python 2.  Ronald patch LGTM.
msg191061 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-06-13 06:32
LGTM.
msg191072 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-13 10:40
On Thu, Jun 13, 2013 at 9:20 AM, Ezio Melotti <report@bugs.python.org>wrote:

>
> Also in the Python 3 docs we don't compare the current behavior with
> Python 2.
>

That's most unfortunate. Major PITA comes from attempts to port existing
code.
msg191073 - (view) Author: anatoly techtonik (techtonik) Date: 2013-06-13 10:41
_failed_ attempts to port existing code.
msg191079 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-06-13 11:16
Surely there are already good places to help with 2->3 transition?
The Library Reference is not such a place.
msg192387 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-06 08:25
New changeset a2c2ffa1a41c by Ronald Oussoren in branch '3.3':
Issue #17860: explicitly mention that std* streams are opened in binary mode by default.
http://hg.python.org/cpython/rev/a2c2ffa1a41c

New changeset ae8b054155c1 by Ronald Oussoren in branch 'default':
(3.3->default) Issue #17860: explicitly mention that std* streams are opened in binary mode by default.
http://hg.python.org/cpython/rev/ae8b054155c1
msg192388 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-06 08:26
The documentation for 3.3 and default now explicitly mentions that the std* streams are opened as binary streams when universal_newlines is false.
msg192516 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-07-07 07:28
New changeset f4747e1ce2b1 by Ronald Oussoren in branch '3.3':
Cleanup of documentation change from #17860
http://hg.python.org/cpython/rev/f4747e1ce2b1

New changeset 83810f67d16b by Ronald Oussoren in branch 'default':
(3.3->default) Cleanup of documentation change from #17860
http://hg.python.org/cpython/rev/83810f67d16b
History
Date User Action Args
2013-07-07 07:28:15python-devsetmessages: + msg192516
2013-07-06 08:26:55ronaldoussorensetstatus: open -> closed
resolution: fixed
messages: + msg192388

stage: commit review -> resolved
2013-07-06 08:25:18python-devsetnosy: + python-dev
messages: + msg192387
2013-06-13 11:16:33amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg191079
2013-06-13 10:41:38techtoniksetmessages: + msg191073
2013-06-13 10:40:42techtoniksetmessages: + msg191072
2013-06-13 08:50:55ronaldoussorensetassignee: docs@python -> ronaldoussoren
stage: needs patch -> commit review
2013-06-13 06:32:14serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg191061
2013-06-13 06:20:37ezio.melottisetmessages: + msg191059
2013-06-13 06:02:19ronaldoussorensetmessages: + msg191058
2013-06-12 19:48:48techtoniksetmessages: + msg191045
2013-06-12 14:57:31ronaldoussorensetfiles: + issue17860.txt
nosy: + ronaldoussoren
messages: + msg191032

2013-05-03 16:38:43tshepangsetnosy: + tshepang
2013-04-28 21:21:26r.david.murraysetnosy: + r.david.murray
messages: + msg188025
2013-04-28 19:19:00techtoniksetmessages: + msg188016
2013-04-28 16:14:45asvetlovsetnosy: + asvetlov
messages: + msg187996
2013-04-28 14:39:13ezio.melottisetnosy: + vstinner, ezio.melotti
stage: needs patch
type: enhancement

versions: + Python 3.4
2013-04-28 14:36:39techtonikcreate