I recently switched to Ubuntu 11.04 from OpenSUSE 11.4, and when I go to run a project I coded a couple days ago under OpenSUSE using the multiprocessing library, it hangs when it did not under OpenSUSE.

Specifically, I am using two queues, work_queue from which the children get jobs, and results_queue where they place their results before calling JoinableQueue.task_done() and grabbing the next result. I use the "poison pill" technique to terminate the children, where a None object is placed at the end of the queue for each child, and when they get one of the terminating objects they call task_done() again (to account for the None object) and exit.

In the main process, after spawning all of the children (one per physical CPU), it joins with the work_queue in order to wait for all of its children to finish.

This is pretty much a cookie-cutter multiprocessing implementation that I've used successfully for years under OpenSUSE, but for some odd reason the exact same code does not work under Ubuntu.

I would try porting to python 3.x, but the rest of my research team is still using 2.7, so that's not really an option right now.

Edit: Sorry, I should have been more clear. The hang occurs after the first child process exits, at which point all four children become zombies (none of the others exit, they just zombify immediately), and the main process sits there waiting forever for the rest of the children to clear out the queue, which of course never happens.

Michael,

It is hard to tell from your description alone where the bug is.  Could you provide more detailed reproduction steps with a test case that exhibits the issue?

Okay, I have attached the code I've been using. Don't worry about what it does (it's a biology thing), but just follow these steps:

1. Make sure you have numpy and scipy installed.
2. Extract the zip file.
3. Run it with ./svm_main.py test_obligate.dat test_transient.dat

The method svm_main.grid_search and the module grid_search_process are probably the only things you need pay attention to, everything else is problem-specific.

I tried switching from joining on the work_queue to just joining on the individual child processes, and it seems to work now. Weird. Anyway, it'd be nice to see the JoinableQueue fixed, but it's not pressing any more.