classification
Title: subprocess: optional auto-reaping fixing os.wait() lossage
Type: Stage:
Components: Library (Lib) Versions: Python 2.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Justin.Lebar, loewis, yorick
Priority: normal Keywords: patch

Created on 2005-04-21 11:40 by yorick, last changed 2020-03-26 17:08 by Justin.Lebar. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess.patch yorick, 2005-04-22 16:20 subprocess.patch
Messages (5)
msg48250 - (view) Author: Mattias Engdegård (yorick) Date: 2005-04-21 11:40
The subprocess module automatically reaps child
processes. It maintains a list of Popen instances, and
each time a new Popen is created, the list is traversed
and a non-polling wait is done for each instance.

I discussed this with the author, Peter Åstrand, and
this behaviour was inherited from the older popen2
code, and is intended to avoid a limitless accretion of
zombies when the user does not take care to wait for
the processes.

However, the auto-reaping interacts badly with
os.wait()/waitpid() since the user is not aware that
the module is reaping children behind her back. In
particular, os.wait(), which is very useful when a
blocking wait for many children is desired, may not
work at all, which caused me to look at the problem in
the first case.

The solution is to allow the user to create Popen
instances that are not auto-reaped. The interface is
otherwise unchanged, and existing code will see no
change in behaviour.

This patch does three things:
- Adds an autoreap parameter to the Popen constructor,
defaulting to True (the previous behaviour)
- Documents the auto-reaper and its interaction with
os.wait()/waitpid(), which was previously missing
- Changes the list of instances to a set, to avoid O(N)
element removal.

For completeness, here is a test case:

import os, subprocess, time
p = subprocess.Popen(["/bin/true"]).pid
time.sleep(1)
subprocess.call(["/bin/false"])
(pid, status) = os.wait()
print "got", pid, "expected", p

The above code will throw an exception. With the patch,
it will work as expected if autoreap=False is added to
the Popen call.
msg48251 - (view) Author: Mattias Engdegård (yorick) Date: 2005-04-21 11:48
Logged In: YES 
user_id=432579

>and a non-polling wait is done for each instance.

Sorry, this should be "non-blocking wait".
msg48252 - (view) Author: Mattias Engdegård (yorick) Date: 2005-04-22 16:20
Logged In: YES 
user_id=432579

Revised patch, using a dict instead of a set (for
compatibility with python 2.2, following PEP 291), and
rename autoreap parameter to "autowait", after discussion
with Peter Åstrand.
msg48253 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2006-04-10 16:03
Logged In: YES 
user_id=21627

This has been fixed in the subversion trunk in a different
way: processes are added to _active only inside __del__. So
as long as the application keeps a reference to the
subprocess object, it can wait for it; auto-reaping only
starts when the last reference was dropped. If you still see
a problem in that approach, please submit a new patch
(relative to svn trunk, preferably).

Marking this one as outdated.
msg365096 - (view) Author: Justin Lebar (Justin.Lebar) Date: 2020-03-26 17:08
15 years later, it seems asyncio subprocesses may have the same issue.  :)

https://bugs.python.org/issue40078
History
Date User Action Args
2020-03-26 17:08:48Justin.Lebarsetnosy: + Justin.Lebar
messages: + msg365096
2005-04-21 11:40:17yorickcreate