I was trying to use subprocess to run multiple processes, and then wait
until one was finished.  I was using poll() to do this and created the
following test case:
#BEGIN
import subprocess,os

procs = [subprocess.Popen(["sleep",str(x)]) for x in range(1,11)]

while len(procs) > 0:
    os.wait()
    print [(p.pid,p.poll()) for p in procs]
    procs = [p for p in procs if p.poll() == None]
#END

I would have expected that as this program was run, it would remove the
processes that finished from the procs list, but instead, they stay in
it and I got the following output:

#Output
[(7426, None), (7427, None), (7428, None), (7429, None), (7430, None),
(7431, None), (7432, None), (7433, None), (7434, None), (7435, None)]
#above line repeats 8 more times
[(7426, None), (7427, None), (7428, None), (7429, None), (7430, None),
(7431, None), (7432, None), (7433, None), (7434, None), (7435, None)]
Traceback (most recent call last):
  File "./test_poll.py", line 9, in <module>
    os.wait()
OSError: [Errno 10] No child processes
#End output

Basically, even for finished processes, poll returns None.

Version of python used:
Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26) 
[GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2

Relevant documentation in Library reference manual 17.1.2
poll(  	) ... Returns returncode attribute.
... A None value indicates that the process hasn't terminated yet.

The problem is that os.wait() does not play nicely with subprocess.py.
Popen.poll() and Popen.wait() use os.waitpid(pid, ...)  which will
raise OSError if pid has already been reported by os.wait().
Popen.poll() swallows OSError and by default returns None.

You can (sort of) fix your program by using
"p.popen(_deadstate='dead')" in place of "p.popen()".  This will make
poll() return 'dead' instead of None if OSError gets caught, but this
is undocumented.

Maybe a subprocess.wait() function could be added which would return a
tuple

    (pid, exitcode, popen_object)

where popen_object is None if the process is "foreign" (i.e. it was
not created by the subprocess module).

It would not be hard to implement this on unix if you don't care about
thread safety.  (On unix Popen.wait() is not thread-safe either, so
maybe thread safety does not matter.)

To implement something similar on windows you would probably need to
use WaitForMultipleObjects() to check whether any process handles are
signalled, but that would involve patching _subprocess.c or using
ctypes or pywin32.

Hm.  Well, after filing the bug, I created a thread for each subprocess,
and had that thread do an wait on the process, and that worked fine. 
So, I guess at minimum it sounds like the documentation for poll could
be improved to mention that it will not catch the state if something
else does.  I think a better fix would be for poll to return some kind
of UnknownError instead of None if the process was finished, but python
did not catch it for some reason (like using os.wait() :)

Isn't this a critical problem. The .poll() function serves as a means to
check the status of the process started. When it continues to report
'None' to a process which has already terminated, it creates a false
positive of a hung process. Dealing with recovery from an actual hung
process is difficult enough. Having to deal with a bad detection that
the process ran to completion on top of this, makes the use of
subprocess difficult. 

Maybe I'm miss applying the .poll() function. I'm trying to detect that
a process has hung, prior to calling .stdout.readlines(). The
readlines() will hang my python script if the process is hung. Is there
another way I should be doing this?

Thanks, 

Mike

I have also run into this problem.  If you only use p.poll() and never
p.wait(), returncode will always remain None.

roudkerk's workaround doesn't seem to work with the new Popen objects,
at least in python 2.4.  ("unexpected keyword argument '_deadstate'") 
Does anyone have a workaround for subprocess.Popen?  Do I have to switch
to the deprecated popen function(s)?

Should this be closed or is this still a problem in 2.7 (release candidate out now, final soon) or 3.1?

Terry,

I had long since coded around the problem. At this point, I no longer have
the test environment to cause the intermittent conditions of the process
hang. I could code something up, but; your question is the first response to
this bug report since "reiko <j.k.langridge@gmail.com>" on 7/28/2008. I
suggest it can be closed, and would be reopened when encountered again. I
appears nobody is working in that area of code.

On Wed, Jun 16, 2010 at 8:13 PM, Terry J. Reedy <report@bugs.python.org>wrote:

>
> Terry J. Reedy <tjreedy@udel.edu> added the comment:
>
> Should this be closed or is this still a problem in 2.7 (release candidate
> out now, final soon) or 3.1?
>
> ----------
> nosy: +tjreedy
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue2475>
> _______________________________________
>