While running a complex python process that executes a bunch of subprocesses (using the subprocess module, specifically calling communicate()), I found myself with occasional zombie processes piling up. Turns out Python is not correctly wait()ing for the children. Although in my case it happens for < 5% of subprocesses, it happens for random Popen objects, used in different ways (using Popen() and then read()/write()/wait() directly or with communicate()). I'd love to find out I'm crazy, but I'm not doing anything too sneaky and the patch below fixes the problem.

I'm not sure why it's happening in my particular environment (maybe it just so happens that the child processes enter into states with particular timing, or the parent receives signals at the wrong moments) but it's very reproducible for me.

I believe that the cause of the zombie processes is as follows:

If you read the description of the waitpid system call (http://www.kernel.org/doc/man-pages/online/pages/man2/wait.2.html), there are several events that could cause waitpid() to return. I have no idea why, but even without WNOHANG set, it looks I'm getting back an occasional 0 return value from waitpid(). Interrupted system call? Stopped child process? Not sure why at the moment. The documentation is a bit ambiguous as to whether this can happen, BUT looking at the example code at the bottom, it seems to handle this spurious wakeup case (which subprocess does not). The net result is that this process has *not* exited or been killed. The python code paths don't consider this possibility (as I believe in normal circumstances, it rarely happens!).

I discovered this bug on 2.7.2. I've prepared a patch for the 2.7 branch (75701:d46c1973d3c4), although I'm certain almost all versions, including the tip suffer from this problem. I'm happy to port to other branches if necessary, although I think appropriate maintainers could whip it up in no time flat. I've tested my 2.7 fix and it solves my problem -- no more zombies. This patch does not change the behaviour of the Popen class in the normal case but allows it to handle spurious wakeups.

Thanks.  I'll see that this fix gets into 2.7, 3.2 and 3.3.

Out of curiosity, what Linux kernel version and glibc version were you using?

I'm somewhat surprised that I haven't run into this before. :)

Kernel is 3.0.0-15-generic (I believe stock Ubuntu Oneric kernel). 
Version for glibc is Ubuntu EGLIBC 2.13-20ubuntu5.

I'm working on figuring out the exact conditions under which it happens and creating a harness. I'll post it when I've got it.

> I'm working on figuring out the exact conditions under which it 
> happens and creating a harness. I'll post it when I've got it.

Please do so, because I'm quite skeptical about waitpid() returning 0
without WNOHANG.
If you can reproduce it fairly consistently, you could try running
under strace to see what's happening.

New changeset d478df13abde by Gregory P. Smith in branch '3.2':
Fixes issue #14396: Handle the odd rare case of waitpid returning 0 when
http://hg.python.org/cpython/rev/d478df13abde

New changeset 61a0eace0f2e by Gregory P. Smith in branch '3.3':
Fixes issue #14396: Handle the odd rare case of waitpid returning 0
http://hg.python.org/cpython/rev/61a0eace0f2e

New changeset 512c1120332f by Gregory P. Smith in branch 'default':
Fixes issue #14396: Handle the odd rare case of waitpid returning 0
http://hg.python.org/cpython/rev/512c1120332f

New changeset 82711f5ab507 by Gregory P. Smith in branch '2.7':
Fixes issue #14396: Handle the odd rare case of waitpid returning 0
http://hg.python.org/cpython/rev/82711f5ab507

regardless of knowing how to reproduce this system call behavior, the changes necessary to handle robustly it are easy enough.  fixed.

3.3+ already handled it if a timeout was specified (new feature).  I only had to fix the default no timeout case.