The documentation in Python 2.6 claims os.popen is deprecated [1]. This
is probably a false claim. I think that at some point os.popen got
confused with os.popen2, os.popen3, os.popen4 and the popen2 module, all
of which are correctly deprecated.

I bring three forward three pieces of evidence that os.popen is *not*
deprecated:

i) In Python 3.1, unlike the others, os.popen is still present and is
not even deprecated. [2]

ii) Unlike the others, using os.popen does not raise a DeprecationWarning;

iii) Unlike the others, its implementation is lifted straight from the
posix module.

--
[1] http://docs.python.org/2.6/library/os.html#os.popen
[2] http://docs.python.org/3.1/library/os.html#os.plock

Actually popen() isn't documented anymore in the 3.1 docs.  However, I
don't know about the "real" deprecation status.  Benjamin?

I have no idea either. It seems os.popen is now a hacked up to use
subprocess, so it seems intentional to keep it. Guido, you made this
change; is os.peopen supposed to be gone in 3.x?

I am guessing the reason to keep os.popen() (albeit now reimplemented 
using subprocess) is that it is a convenient wrapper for a common use case 
and also familiar.  I see no problem with this.  (Indeed the big problem 
was with the proliferation of popenN with confusing signatures.)  So I 
guess it ought to be documented and removed from the list of deprecations 
in 2.6.

By the way, shouldn't the various posix.spawnv* functions be officiall
deprecated as well?

Georg, please note that os.popen *is* documented in 3.1. See attached
screen shot.

That is not really a documentation for the function, but a pointer to a
section in which there is no documentation for popen.

The function is still marked as deprecated in the 2.x doc and undocumented in 3.x (the function is there, but the description points somewhere else).

The attached patch grabs the os.popen documentation from python 2.7, throws away the deprecation notes, and adds a pointer to subprocess.Popen for more details.

I will refresh the patch, update it to recommend use as a context manager, and submit the patch here for review before committing.  It’s too late for 2.6, though.

Benjamin, I hope you won’t mind me taking the assignment from you.

r55334 removed popen2, popen3 and popen4 from the os module from Python 3 (before the 3.0 release), but not os.popen.

Python 3.2 has now convenience functions in subprocess to get the output of a program:

- check_output()
- getstatusoutput()
- getoutput()

They have a better API then os.popen because you can:

 - avoid the shell process and use a list of arguments (to avoid ugly shell quoting using ' or ")
 - change the current directory
 - decide if signal are restored or not
 - get easily the exit code (not >> 8 magical operation on the close() output)
 - use a different executable name
 - pass file descriptor
 - etc.

I think that it's time to move forward and remove os.popen(). But... there are 66 calls to os.popen() in:

Doc/tools/docutils/writers/odf_odt
Lib
Lib/ctypes
Lib/distutils
Lib/distutils/command
Lib/distutils/tests
Lib/idlelib
Lib/multiprocessing
Lib/pydoc_data
Lib/test
Lib/tkinter/test/test_tkinter
Mac/BuildScript
Mac/Tools
PCbuild
PC/VC6
PC/VS7.1
PC/VS8.0
Tools/msi
Tools/scripts

So we can maybe start with:
 - Restore the documentation
 - Mark this function as deprecated in the doc
 - Emit a deprecating warning in the code
 - Start to replace os.popen() with subprocess in Python

And then maybe never drop it :-)

Note: even subprocess.getstatusoutput() is implemented using os.popen()...

The following documentation should also be updated to use the new convenience functions:
http://docs.python.org/py3k/library/subprocess.html#replacing-os-popen-os-popen2-os-popen3

STINNER Victor wrote:
> 
> STINNER Victor <victor.stinner@haypocalc.com> added the comment:
> 
> r55334 removed popen2, popen3 and popen4 from the os module from Python 3 (before the 3.0 release), but not os.popen.
> 
> Python 3.2 has now convenience functions in subprocess to get the output of a program:
> 
> - check_output()
> - getstatusoutput()
> - getoutput()
> 
> They have a better API then os.popen because you can:
> 
>  - avoid the shell process and use a list of arguments (to avoid ugly shell quoting using ' or ")
>  - change the current directory
>  - decide if signal are restored or not
>  - get easily the exit code (not >> 8 magical operation on the close() output)
>  - use a different executable name
>  - pass file descriptor
>  - etc.

Right, but not all applications need all those features and
os.popen() has a nice, simple and well-known interface.

Since os.popen() is already implemented using subprocess, all
applications using os.popen() will benefit from any bug fixes
immediately, so the support effort in keeping it around is very low.

> I think that it's time to move forward and remove os.popen(). But... there are 66 calls to os.popen() in:
> 
> Doc/tools/docutils/writers/odf_odt
> Lib
> Lib/ctypes
> Lib/distutils
> Lib/distutils/command
> Lib/distutils/tests
> Lib/idlelib
> Lib/multiprocessing
> Lib/pydoc_data
> Lib/test
> Lib/tkinter/test/test_tkinter
> Mac/BuildScript
> Mac/Tools
> PCbuild
> PC/VC6
> PC/VS7.1
> PC/VS8.0
> Tools/msi
> Tools/scripts
> 
> So we can maybe start with:
>  - Restore the documentation

+1

As for the other popen() variants: those are already gone and are
indeed better implemented using subprocess directly.

Per msg129958, attached is my stab at a patch to replace most uses of os.popen() with the subprocess module. The test suite passes on my Mac, but the patch does touch some specific-to-other-platform code, so further testing is obviously needed.
This is my first non-docs patch, please be gentle. :) [Those patches were to subprocess' docs though!]

Stuff still using os.popen() that the patch doesn't fix:
- multiprocessing
- platform.popen() [which is itself deprecated]
- subprocess.check_output()
- Lib/test/test_poll.py
- Lib/test/test_select.py
- Lib/distutils/tests/test_cygwinccompiler.py

Also, I suppose Issue 9382 should be marked as a dupe of this one?

Chris Rebert wrote:
> 
> Chris Rebert <pybugs@rebertia.com> added the comment:
> 
> Per msg129958, attached is my stab at a patch to replace most uses of os.popen() with the subprocess module. The test suite passes on my Mac, but the patch does touch some specific-to-other-platform code, so further testing is obviously needed.
> This is my first non-docs patch, please be gentle. :) [Those patches were to subprocess' docs though!]
> 
> Stuff still using os.popen() that the patch doesn't fix:
> - multiprocessing
> - platform.popen() [which is itself deprecated]
> - subprocess.check_output()
> - Lib/test/test_poll.py
> - Lib/test/test_select.py
> - Lib/distutils/tests/test_cygwinccompiler.py
> 
> Also, I suppose Issue 9382 should be marked as a dupe of this one?

Thanks, but I still don't understand why os.popen() wasn't removed
from the list of deprecated APIs as per Guido's message further up
on the ticket.

If you look at the amount of code you need to add in order
to support the os.popen() functionality directly using
subprocess instead of going the indirect way via the existing
os.popen() wrapper around the subprocess functionality, I think
this shows that the wrapper is indeed a good thing to have
and something you'd otherwise implement anyway as part of
standard code refactoring.

So instead of applying such a patch, I think we should add back
the documentation for os.popen() and remove the deprecation
notices.

The deprecations for os.popenN() are still fine, since those
APIs are not used all that much, and I'm sure that no one can
really remember what all the different versions do anyway :-)

os.popen() OTOH is often used and implements a very common
use: running an external command and getting the stdout
results back for further processing.

> The deprecations for os.popenN() are still fine, since those
> APIs are not used all that much, and I'm sure that no one can
> really remember what all the different versions do anyway :-)

That's good, because those functions are already gone in Python3 :)

Victor: did you notice that getoutput and friends call os.popen?

The one argument I can see for actually deprecating os.popen is that in general the os module functions are thin wrappers around the corresponding posix functions.  But that is only true in general, and os.popen was never a "thin" wrapper, at least on most platforms.

> Victor: did you notice that getoutput and friends call os.popen?

Yes, because I wrote a patch to call directly subprocess :-) => see the issue 
#10197. I don't want to remove os.popen() anymore, it's too much work for a 
minor gain (we will remove it in Python 4). But we can add a note in its 
documentation saying that the subprocess should be prefered.

> But that is only true in general, and os.popen was never
> a "thin" wrapper, at least on most platforms.

os.popen() was a thin wrapper in Python 2.x: posix.popen() was a wrapper of 
the popen() C function.

There were an awfully large number of pages of code for a thin wrapper.  Granted, half of that was probably os/2 support...

Please apply Neil Muller’s documentation patch. It is certainly better than the current state.

If you want to improve it further, maybe get rid of the trailing comma, and mention that the close() method returns the exit status encoded like the wait() function.

Also it would be good to document that it returns a text stream, not a binary stream.

Updated the patch as per Martin's notes in msg203483.

I thought the discussion of the return code was rather complicated and tried rewriting it; new patch attached.  Is it an improvement?  The new version also specifies the function's parameters.

Yes, I think that wording works a lot better. Thanks for the touch-up.

New changeset 3417a95df7e2 by Andrew Kuchling in branch 'default':
#6490: Expand documentation for os.popen().
http://hg.python.org/cpython/rev/3417a95df7e2

Thanks for your patch!