classification
Title: os.popen documentation is probably wrong
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.1, Python 3.2, Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: Neil Muller, akuchling, amaury.forgeotdarc, benjamin.peterson, cvrebert, eric.araujo, ezio.melotti, georg.brandl, gvanrossum, krawyoti, lemburg, martin.panter, python-dev, r.david.murray, sam.kimbrel, vstinner
Priority: normal Keywords: patch

Created on 2009-07-15 08:35 by krawyoti, last changed 2014-04-16 13:11 by akuchling. This issue is now closed.

Files
File name Uploaded Description Edit
docs.png krawyoti, 2009-07-16 21:18 Screen shot from the 3.1 docs
os_popen_doc.diff Neil Muller, 2010-11-20 15:07 Redocument os.popen, using docs from python 2.7 review
mostly_replace_os_popen.patch cvrebert, 2011-05-30 10:57 Replaces most of uses of os.popen() with subprocess review
6490-os-popen-docs.diff sam.kimbrel, 2014-04-15 20:53 review
6490-os-popen-docs.diff-2.txt akuchling, 2014-04-16 02:48 review
Messages (25)
msg90532 - (view) Author: krawyoti (krawyoti) Date: 2009-07-15 08:35
The documentation in Python 2.6 claims os.popen is deprecated [1]. This
is probably a false claim. I think that at some point os.popen got
confused with os.popen2, os.popen3, os.popen4 and the popen2 module, all
of which are correctly deprecated.

I bring three forward three pieces of evidence that os.popen is *not*
deprecated:

i) In Python 3.1, unlike the others, os.popen is still present and is
not even deprecated. [2]

ii) Unlike the others, using os.popen does not raise a DeprecationWarning;

iii) Unlike the others, its implementation is lifted straight from the
posix module.

--
[1] http://docs.python.org/2.6/library/os.html#os.popen
[2] http://docs.python.org/3.1/library/os.html#os.plock
msg90558 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-07-16 07:30
Actually popen() isn't documented anymore in the 3.1 docs.  However, I
don't know about the "real" deprecation status.  Benjamin?
msg90569 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2009-07-16 12:54
I have no idea either. It seems os.popen is now a hacked up to use
subprocess, so it seems intentional to keep it. Guido, you made this
change; is os.peopen supposed to be gone in 3.x?
msg90572 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2009-07-16 14:48
I am guessing the reason to keep os.popen() (albeit now reimplemented 
using subprocess) is that it is a convenient wrapper for a common use case 
and also familiar.  I see no problem with this.  (Indeed the big problem 
was with the proliferation of popenN with confusing signatures.)  So I 
guess it ought to be documented and removed from the list of deprecations 
in 2.6.
msg90573 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2009-07-16 14:52
By the way, shouldn't the various posix.spawnv* functions be officiall
deprecated as well?
msg90586 - (view) Author: krawyoti (krawyoti) Date: 2009-07-16 21:18
Georg, please note that os.popen *is* documented in 3.1. See attached
screen shot.
msg90589 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-07-16 21:46
That is not really a documentation for the function, but a pointer to a
section in which there is no documentation for popen.
msg98498 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2010-01-29 02:41
The function is still marked as deprecated in the 2.x doc and undocumented in 3.x (the function is there, but the description points somewhere else).
msg121660 - (view) Author: Neil Muller (Neil Muller) Date: 2010-11-20 15:07
The attached patch grabs the os.popen documentation from python 2.7, throws away the deprecation notes, and adds a pointer to subprocess.Popen for more details.
msg123429 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-12-05 18:00
I will refresh the patch, update it to recommend use as a context manager, and submit the patch here for review before committing.  It’s too late for 2.6, though.

Benjamin, I hope you won’t mind me taking the assignment from you.
msg129958 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-03-03 11:56
r55334 removed popen2, popen3 and popen4 from the os module from Python 3 (before the 3.0 release), but not os.popen.

Python 3.2 has now convenience functions in subprocess to get the output of a program:

- check_output()
- getstatusoutput()
- getoutput()

They have a better API then os.popen because you can:

 - avoid the shell process and use a list of arguments (to avoid ugly shell quoting using ' or ")
 - change the current directory
 - decide if signal are restored or not
 - get easily the exit code (not >> 8 magical operation on the close() output)
 - use a different executable name
 - pass file descriptor
 - etc.

I think that it's time to move forward and remove os.popen(). But... there are 66 calls to os.popen() in:

Doc/tools/docutils/writers/odf_odt
Lib
Lib/ctypes
Lib/distutils
Lib/distutils/command
Lib/distutils/tests
Lib/idlelib
Lib/multiprocessing
Lib/pydoc_data
Lib/test
Lib/tkinter/test/test_tkinter
Mac/BuildScript
Mac/Tools
PCbuild
PC/VC6
PC/VS7.1
PC/VS8.0
Tools/msi
Tools/scripts

So we can maybe start with:
 - Restore the documentation
 - Mark this function as deprecated in the doc
 - Emit a deprecating warning in the code
 - Start to replace os.popen() with subprocess in Python

And then maybe never drop it :-)

Note: even subprocess.getstatusoutput() is implemented using os.popen()...
msg129959 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-03-03 11:57
The following documentation should also be updated to use the new convenience functions:
http://docs.python.org/py3k/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3
msg130448 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-03-09 11:24
STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> r55334 removed popen2, popen3 and popen4 from the os module from Python 3 (before the 3.0 release), but not os.popen.
> 
> Python 3.2 has now convenience functions in subprocess to get the output of a program:
> 
> - check_output()
> - getstatusoutput()
> - getoutput()
> 
> They have a better API then os.popen because you can:
> 
>  - avoid the shell process and use a list of arguments (to avoid ugly shell quoting using ' or ")
>  - change the current directory
>  - decide if signal are restored or not
>  - get easily the exit code (not >> 8 magical operation on the close() output)
>  - use a different executable name
>  - pass file descriptor
>  - etc.

Right, but not all applications need all those features and
os.popen() has a nice, simple and well-known interface.

Since os.popen() is already implemented using subprocess, all
applications using os.popen() will benefit from any bug fixes
immediately, so the support effort in keeping it around is very low.

> I think that it's time to move forward and remove os.popen(). But... there are 66 calls to os.popen() in:
> 
> Doc/tools/docutils/writers/odf_odt
> Lib
> Lib/ctypes
> Lib/distutils
> Lib/distutils/command
> Lib/distutils/tests
> Lib/idlelib
> Lib/multiprocessing
> Lib/pydoc_data
> Lib/test
> Lib/tkinter/test/test_tkinter
> Mac/BuildScript
> Mac/Tools
> PCbuild
> PC/VC6
> PC/VS7.1
> PC/VS8.0
> Tools/msi
> Tools/scripts
> 
> So we can maybe start with:
>  - Restore the documentation

+1

As for the other popen() variants: those are already gone and are
indeed better implemented using subprocess directly.
msg137255 - (view) Author: Chris Rebert (cvrebert) * Date: 2011-05-30 10:57
Per msg129958, attached is my stab at a patch to replace most uses of os.popen() with the subprocess module. The test suite passes on my Mac, but the patch does touch some specific-to-other-platform code, so further testing is obviously needed.
This is my first non-docs patch, please be gentle. :) [Those patches were to subprocess' docs though!]

Stuff still using os.popen() that the patch doesn't fix:
- multiprocessing
- platform.popen() [which is itself deprecated]
- subprocess.check_output()
- Lib/test/test_poll.py
- Lib/test/test_select.py
- Lib/distutils/tests/test_cygwinccompiler.py

Also, I suppose Issue 9382 should be marked as a dupe of this one?
msg137256 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2011-05-30 11:27
Chris Rebert wrote:
> 
> Chris Rebert <pybugs@rebertia.com> added the comment:
> 
> Per msg129958, attached is my stab at a patch to replace most uses of os.popen() with the subprocess module. The test suite passes on my Mac, but the patch does touch some specific-to-other-platform code, so further testing is obviously needed.
> This is my first non-docs patch, please be gentle. :) [Those patches were to subprocess' docs though!]
> 
> Stuff still using os.popen() that the patch doesn't fix:
> - multiprocessing
> - platform.popen() [which is itself deprecated]
> - subprocess.check_output()
> - Lib/test/test_poll.py
> - Lib/test/test_select.py
> - Lib/distutils/tests/test_cygwinccompiler.py
> 
> Also, I suppose Issue 9382 should be marked as a dupe of this one?

Thanks, but I still don't understand why os.popen() wasn't removed
from the list of deprecated APIs as per Guido's message further up
on the ticket.

If you look at the amount of code you need to add in order
to support the os.popen() functionality directly using
subprocess instead of going the indirect way via the existing
os.popen() wrapper around the subprocess functionality, I think
this shows that the wrapper is indeed a good thing to have
and something you'd otherwise implement anyway as part of
standard code refactoring.

So instead of applying such a patch, I think we should add back
the documentation for os.popen() and remove the deprecation
notices.

The deprecations for os.popenN() are still fine, since those
APIs are not used all that much, and I'm sure that no one can
really remember what all the different versions do anyway :-)

os.popen() OTOH is often used and implements a very common
use: running an external command and getting the stdout
results back for further processing.
msg137297 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-30 15:40
> The deprecations for os.popenN() are still fine, since those
> APIs are not used all that much, and I'm sure that no one can
> really remember what all the different versions do anyway :-)

That's good, because those functions are already gone in Python3 :)

Victor: did you notice that getoutput and friends call os.popen?

The one argument I can see for actually deprecating os.popen is that in general the os module functions are thin wrappers around the corresponding posix functions.  But that is only true in general, and os.popen was never a "thin" wrapper, at least on most platforms.
msg137299 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2011-05-30 15:52
> Victor: did you notice that getoutput and friends call os.popen?

Yes, because I wrote a patch to call directly subprocess :-) => see the issue 
#10197. I don't want to remove os.popen() anymore, it's too much work for a 
minor gain (we will remove it in Python 4). But we can add a note in its 
documentation saying that the subprocess should be prefered.

> But that is only true in general, and os.popen was never
> a "thin" wrapper, at least on most platforms.

os.popen() was a thin wrapper in Python 2.x: posix.popen() was a wrapper of 
the popen() C function.
msg137302 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-05-30 16:44
There were an awfully large number of pages of code for a thin wrapper.  Granted, half of that was probably os/2 support...
msg203483 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2013-11-20 14:32
Please apply Neil Muller’s documentation patch. It is certainly better than the current state.

If you want to improve it further, maybe get rid of the trailing comma, and mention that the close() method returns the exit status encoded like the wait() function.
msg203557 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2013-11-21 01:57
Also it would be good to document that it returns a text stream, not a binary stream.
msg216384 - (view) Author: Sam Kimbrel (sam.kimbrel) * Date: 2014-04-15 20:53
Updated the patch as per Martin's notes in msg203483.
msg216424 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2014-04-16 02:48
I thought the discussion of the return code was rather complicated and tried rewriting it; new patch attached.  Is it an improvement?  The new version also specifies the function's parameters.
msg216426 - (view) Author: Sam Kimbrel (sam.kimbrel) * Date: 2014-04-16 02:52
Yes, I think that wording works a lot better. Thanks for the touch-up.
msg216461 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-04-16 13:11
New changeset 3417a95df7e2 by Andrew Kuchling in branch 'default':
#6490: Expand documentation for os.popen().
http://hg.python.org/cpython/rev/3417a95df7e2
msg216462 - (view) Author: A.M. Kuchling (akuchling) * (Python committer) Date: 2014-04-16 13:11
Thanks for your patch!
History
Date User Action Args
2014-04-16 13:11:39akuchlingsetstatus: open -> closed
title: os.popen documentation is probably wrong -> os.popen documentation is probably wrong
messages: + msg216462

resolution: fixed
stage: needs patch -> resolved
2014-04-16 13:11:08python-devsetnosy: + python-dev
messages: + msg216461
2014-04-16 02:52:16sam.kimbrelsetmessages: + msg216426
2014-04-16 02:48:22akuchlingsetfiles: + 6490-os-popen-docs.diff-2.txt
nosy: + akuchling
messages: + msg216424

2014-04-15 20:53:06sam.kimbrelsetfiles: + 6490-os-popen-docs.diff
nosy: + sam.kimbrel
messages: + msg216384

2013-11-21 01:57:22martin.pantersetmessages: + msg203557
2013-11-20 14:32:09martin.pantersetnosy: + martin.panter

messages: + msg203483
versions: + Python 3.3, Python 3.4
2012-08-08 17:31:44r.david.murraylinkissue15585 superseder
2012-08-08 17:29:59r.david.murraylinkissue15584 superseder
2011-06-04 00:19:49terry.reedysettitle: os.popen documentation in 2.6 is probably wrong -> os.popen documentation is probably wrong
2011-05-30 16:44:29r.david.murraysetmessages: + msg137302
2011-05-30 15:52:31vstinnersetmessages: + msg137299
2011-05-30 15:40:09r.david.murraysetnosy: + r.david.murray
messages: + msg137297
2011-05-30 15:29:52eric.araujolinkissue9382 superseder
2011-05-30 11:27:32lemburgsetmessages: + msg137256
2011-05-30 10:57:28cvrebertsetfiles: + mostly_replace_os_popen.patch
nosy: + cvrebert
messages: + msg137255

2011-03-09 11:24:06lemburgsetnosy: + lemburg
messages: + msg130448
2011-03-03 11:57:46vstinnersetnosy: gvanrossum, georg.brandl, amaury.forgeotdarc, vstinner, benjamin.peterson, ezio.melotti, Neil Muller, eric.araujo, krawyoti
messages: + msg129959
2011-03-03 11:56:24vstinnersetnosy: + vstinner
messages: + msg129958
2010-12-05 18:00:27eric.araujosetversions: + Python 3.1, - Python 2.6
nosy: + eric.araujo

messages: + msg123429

assignee: benjamin.peterson -> eric.araujo
2010-12-04 15:02:50Neil Mullersetversions: + Python 3.2
2010-11-20 15:08:00Neil Mullersetfiles: + os_popen_doc.diff

nosy: + Neil Muller
messages: + msg121660

keywords: + patch
2010-01-29 02:41:17ezio.melottisetpriority: normal

nosy: + ezio.melotti
messages: + msg98498

stage: needs patch
2009-07-16 21:46:09georg.brandlsetmessages: + msg90589
2009-07-16 21:18:02krawyotisetfiles: + docs.png

messages: + msg90586
2009-07-16 14:52:03amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg90573
2009-07-16 14:48:32gvanrossumsetmessages: + msg90572
2009-07-16 12:54:47benjamin.petersonsetnosy: + gvanrossum
messages: + msg90569
2009-07-16 07:30:20georg.brandlsetassignee: georg.brandl -> benjamin.peterson

messages: + msg90558
nosy: + benjamin.peterson
2009-07-15 08:36:09krawyotisettype: behavior
2009-07-15 08:35:38krawyoticreate