This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author amikoren@yahoo.com
Recipients amikoren@yahoo.com
Date 2015-12-09.10:04:32
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1449655474.19.0.730082771595.issue25829@psf.upfronthosting.co.za>
In-reply-to
Content
Happens on Linux (Debian), Linux version 3.16.0-4-amd64 .
Seems like a multiprocessing issue.

When I use both multiprocessing pool and subprocess somewhere in the same python program, sometimes the subprocess become
'zombie', and the parent wait for it forever.

Reproduce:
 run the attached script (I ran it on both python 3.4 and 3.5), and wait (up to a minute in my
computer). Eventually, the script will hang (wait forever).

After it hangs:
ps -ef | grep "hang_multiprocess\|ls"

You should see now the "[ls] <defunct>" process - zombie.


Analysis:
Players:
- Parent process
- Subprocess Child - forked by parent using subprocess.popen()
- Handle_workers thread - multiprocessing thread responsible for verifying all workers are OK, and create them if not.
- Multiprocessing Worker - forked by multiprocessing, either at handle_workers thread context, or at main thread context.

The problem, in a nutshell, is that Handle_workers thread forks a Worker, while Subprocess Child creation.
This causes one of the Child pipes to be 'copied' to the Worker. When the Subprocess Child finishes, the
pipe is still alive (at Worker), hence Parent Process wait forever for the pipe to finish. Child turn into zombie because Parent doesn't reach the communicate/wait line.

In more details:
- The problematic line at subprocess is at  subprocess.py->_execute_child(), before  'while True:' loop, where errpipe_read pipe is read.
- The entry point at multiprocessing is at multiprocessing/pool.py->_handle_workers(). There the thread sleeps for 0.1,
  and then try to create (=fork) new workers.

Handle_workers thread 'copies' errpipe_read to the forked Worker. Hence the pipe never gets closed.

To me, it seems like a multiprocessing issue: The forking from a thread at multiprocessing module is the cause for this mess.

I'm a newbe at Python (first bug launched), so please be patient if I missed anything or jumped into non-based conclusions.
History
Date User Action Args
2015-12-09 10:04:34amikoren@yahoo.comsetrecipients: + amikoren@yahoo.com
2015-12-09 10:04:34amikoren@yahoo.comsetmessageid: <1449655474.19.0.730082771595.issue25829@psf.upfronthosting.co.za>
2015-12-09 10:04:34amikoren@yahoo.comlinkissue25829 messages
2015-12-09 10:04:32amikoren@yahoo.comcreate