Created on 2008-03-24 20:59 by jjcogliati, last changed 2010-08-06 19:52 by dandrzejewski. This issue is now closed.
|unnamed||lisanke, 2010-06-19 02:15|
|msg64439 - (view)||Author: Josh Cogliati (jjcogliati)||Date: 2008-03-24 20:59|
I was trying to use subprocess to run multiple processes, and then wait until one was finished. I was using poll() to do this and created the following test case: #BEGIN import subprocess,os procs = [subprocess.Popen(["sleep",str(x)]) for x in range(1,11)] while len(procs) > 0: os.wait() print [(p.pid,p.poll()) for p in procs] procs = [p for p in procs if p.poll() == None] #END I would have expected that as this program was run, it would remove the processes that finished from the procs list, but instead, they stay in it and I got the following output: #Output [(7426, None), (7427, None), (7428, None), (7429, None), (7430, None), (7431, None), (7432, None), (7433, None), (7434, None), (7435, None)] #above line repeats 8 more times [(7426, None), (7427, None), (7428, None), (7429, None), (7430, None), (7431, None), (7432, None), (7433, None), (7434, None), (7435, None)] Traceback (most recent call last): File "./test_poll.py", line 9, in <module> os.wait() OSError: [Errno 10] No child processes #End output Basically, even for finished processes, poll returns None. Version of python used: Python 2.5.1 (r251:54863, Oct 30 2007, 13:45:26) [GCC 4.1.2 20070925 (Red Hat 4.1.2-33)] on linux2 Relevant documentation in Library reference manual 17.1.2 poll( ) ... Returns returncode attribute. ... A None value indicates that the process hasn't terminated yet.
|msg64901 - (view)||Author: roudkerk (roudkerk)||Date: 2008-04-03 17:38|
The problem is that os.wait() does not play nicely with subprocess.py. Popen.poll() and Popen.wait() use os.waitpid(pid, ...) which will raise OSError if pid has already been reported by os.wait(). Popen.poll() swallows OSError and by default returns None. You can (sort of) fix your program by using "p.popen(_deadstate='dead')" in place of "p.popen()". This will make poll() return 'dead' instead of None if OSError gets caught, but this is undocumented. Maybe a subprocess.wait() function could be added which would return a tuple (pid, exitcode, popen_object) where popen_object is None if the process is "foreign" (i.e. it was not created by the subprocess module). It would not be hard to implement this on unix if you don't care about thread safety. (On unix Popen.wait() is not thread-safe either, so maybe thread safety does not matter.) To implement something similar on windows you would probably need to use WaitForMultipleObjects() to check whether any process handles are signalled, but that would involve patching _subprocess.c or using ctypes or pywin32.
|msg64911 - (view)||Author: Josh Cogliati (jjcogliati)||Date: 2008-04-04 03:33|
Hm. Well, after filing the bug, I created a thread for each subprocess, and had that thread do an wait on the process, and that worked fine. So, I guess at minimum it sounds like the documentation for poll could be improved to mention that it will not catch the state if something else does. I think a better fix would be for poll to return some kind of UnknownError instead of None if the process was finished, but python did not catch it for some reason (like using os.wait() :)
|msg68228 - (view)||Author: Mike Lisanke (lisanke)||Date: 2008-06-15 02:38|
Isn't this a critical problem. The .poll() function serves as a means to check the status of the process started. When it continues to report 'None' to a process which has already terminated, it creates a false positive of a hung process. Dealing with recovery from an actual hung process is difficult enough. Having to deal with a bad detection that the process ran to completion on top of this, makes the use of subprocess difficult. Maybe I'm miss applying the .poll() function. I'm trying to detect that a process has hung, prior to calling .stdout.readlines(). The readlines() will hang my python script if the process is hung. Is there another way I should be doing this? Thanks, Mike
|msg70347 - (view)||Author: (reiko)||Date: 2008-07-28 13:03|
I have also run into this problem. If you only use p.poll() and never p.wait(), returncode will always remain None. roudkerk's workaround doesn't seem to work with the new Popen objects, at least in python 2.4. ("unexpected keyword argument '_deadstate'") Does anyone have a workaround for subprocess.Popen? Do I have to switch to the deprecated popen function(s)?
|msg107967 - (view)||Author: Terry J. Reedy (terry.reedy) *||Date: 2010-06-17 00:13|
Should this be closed or is this still a problem in 2.7 (release candidate out now, final soon) or 3.1?
|msg108160 - (view)||Author: Mike Lisanke (lisanke)||Date: 2010-06-19 02:15|
Terry, I had long since coded around the problem. At this point, I no longer have the test environment to cause the intermittent conditions of the process hang. I could code something up, but; your question is the first response to this bug report since "reiko <firstname.lastname@example.org>" on 7/28/2008. I suggest it can be closed, and would be reopened when encountered again. I appears nobody is working in that area of code. On Wed, Jun 16, 2010 at 8:13 PM, Terry J. Reedy <email@example.com>wrote: > > Terry J. Reedy <firstname.lastname@example.org> added the comment: > > Should this be closed or is this still a problem in 2.7 (release candidate > out now, final soon) or 3.1? > > ---------- > nosy: +tjreedy > > _______________________________________ > Python tracker <email@example.com> > <http://bugs.python.org/issue2475> > _______________________________________ >
|2010-06-19 03:21:17||terry.reedy||set||status: open -> closed|
resolution: out of date
messages: + msg108160
messages: + msg107967
messages: + msg70347
messages: + msg68228
|2008-04-04 03:33:28||jjcogliati||set||messages: + msg64911|
messages: + msg64901