Title: os.fork/os.popen behaviour change between 2.7 and 3.2
Type: Stage: resolved
Components: Versions: Python 3.3
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: doerwalter, sbt, zach.ware
Priority: normal Keywords:

Created on 2012-07-20 15:56 by doerwalter, last changed 2020-04-26 16:54 by zach.ware. This issue is now closed.

File name Uploaded Description Edit doerwalter, 2012-07-20 15:55
Messages (7)
msg165942 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2012-07-20 15:55
The attached script behaves differently on Python 2.7.2 and Python 3.2.3.

With Python 2.7 the script runs for ca. 30 seconds and then I get back my prompt.

With Python 3.2 the script runs in the background, I get back my prompt immediately and can type shell commands.

Commenting out the call to uname() changes the behaviour of the script on Python 3.2 so that it behaves like on Python 2.7.

(This happens on both Max OS X 10.7 and Linux.)
msg165955 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-20 17:45
The problem is that os.wait() is returning when the wrong process exits.

You can fix this by specifying the pid you are waiting for by doing "os.waitpid(pid, 0)" instead of "os.wait()".

Arguably os.popen() and subprocess.communicate() etc should always reap the pid of the process they started.
msg165963 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-20 18:09
Actually, if you replace



        f = os.popen("uname")


        with os.popen("uname") as f:

then things should work.  This is because f.close() waits for the pid.
msg166212 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-23 08:09
In Python 2.x, when the file object returned by popen() is garbage collected the process is automatically waited on, collecting the pid of the process.

In Python 3.x a wrapper object is used whose close method wait on the pid.  This close method is *not* invoked when the wrapper is garbage collected.  However, the set of dead pids cannot accumulate since dead pids get collected whenever os.popen()/subprocess.Popen() is used.

The old behaviour is only an advantage in a refcounted implementation of Python like CPython.  I am not sure that "fixing" the current behaviour is necessary.

Simply adding to the wrapper class a __del__() method which invokes close() will not work.  (One could instead use a weakref callback, but that is non-trivial.)
msg166215 - (view) Author: Walter Dörwald (doerwalter) * (Python committer) Date: 2012-07-23 09:32
So is this simply a documentation issue, or can we close the bug as "won't fix"?
msg166242 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-07-23 18:20
A program which depends on the old behaviour would be broken on a non-refcounted implementation of Python, so I would be inclined to say "won't fix".

However, I think the following patch would restore the old behaviour

diff -r a970054a93fb Lib/
--- a/Lib/	Mon Jul 16 18:30:03 2012 +0100
+++ b/Lib/	Mon Jul 23 19:12:38 2012 +0100
@@ -1004,6 +1004,13 @@
 # Helper for popen() -- a proxy for a file whose close waits for the process
 class _wrap_close:
     def __init__(self, stream, proc):
+        # proc should be waited on when stream is garbage collected
+        import weakref
+        def callback(wr):
+            proc._weakref = None           # break ref cycle
+            proc.wait()
+        proc._weakref = weakref.ref(stream, callback)
+        proc.stdin = proc.stdout = None    # proc must not keep stream alive
         self._stream = stream
         self._proc = proc
     def close(self):
msg367315 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2020-04-26 16:54
Given that you both nearly agreed to "wont fix" 8 years ago and 2.7 is now EOL, I'm going to go ahead and close the issue that way :)
Date User Action Args
2020-04-26 16:54:07zach.waresetstatus: open -> closed

nosy: + zach.ware
messages: + msg367315

resolution: wont fix
stage: resolved
2012-07-23 18:20:40sbtsetmessages: + msg166242
2012-07-23 09:32:26doerwaltersetmessages: + msg166215
2012-07-23 08:09:49sbtsetmessages: + msg166212
versions: + Python 3.3, - Python 3.2
2012-07-20 18:09:45sbtsetmessages: + msg165963
2012-07-20 17:45:34sbtsetnosy: + sbt
messages: + msg165955
2012-07-20 15:56:00doerwaltercreate