classification
Title: multiprocessing: pool.map hangs on empty list
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0, Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jnoller Nosy List: ede, ezio.melotti, jnoller
Priority: normal Keywords: patch

Created on 2009-07-07 11:39 by ede, last changed 2009-07-16 14:23 by jnoller. This issue is now closed.

Files
File name Uploaded Description Edit
map_testandfix.patch ede, 2009-07-07 11:39 a patch including the testcase and the fix
Messages (6)
msg90228 - (view) Author: Eric Eisner (ede) Date: 2009-07-07 11:39
In multiprocessing, if you give a pool.map a zero-length iterator and
specify a nonzero chunksize, the process hangs indefinitely. Example:

import multiprocessing
pool = multiprocessing.Pool()
pool.map(len, [], chunksize=1)
# hang forever

Attached simple testcase and simple fix. I observed this behavior on 2.6
and 3.1, but only verified the patch on 3.1. Unless the line numbers
changed it will probably fix it on 2.6 as well.
msg90231 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2009-07-07 12:06
I'm not familiar with multiprocessing but I gave a quick look at the
patch. If the 'iterable' is an iterator/generator "if len(iterable) ==
0:" may fail since these objects don't usually have a __len__. More
tests for this situation are probably needed too.
msg90232 - (view) Author: Eric Eisner (ede) Date: 2009-07-07 12:19
A few lines before this patch the code turns iterable into a list if it
does not hasattr('__len__') for the purpose of calculating the chunksize.
msg90483 - (view) Author: Eric Eisner (ede) Date: 2009-07-13 14:08
Can anyone review this patch? It is a very simple fix to catch this one
edge case. As this can cause multiprocessing to enter a state where it
needs to be externally killed, it would be good if this made 2.6.3.
msg90484 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-13 14:25
It's on my list for this week.
msg90571 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-07-16 14:23
committed r74023 on trunk
History
Date User Action Args
2009-07-16 14:23:46jnollersetstatus: open -> closed
resolution: fixed
2009-07-16 14:23:35jnollersetmessages: + msg90571
2009-07-13 14:25:24jnollersetmessages: + msg90484
2009-07-13 14:08:30edesetmessages: + msg90483
2009-07-07 13:13:14jnollersetassignee: jnoller
2009-07-07 13:11:48r.david.murraysetnosy: + jnoller
2009-07-07 12:19:15edesetmessages: + msg90232
2009-07-07 12:06:00ezio.melottisetpriority: normal
versions: + Python 3.2, - Python 2.6
nosy: + ezio.melotti

messages: + msg90231

type: crash -> behavior
2009-07-07 11:39:03edecreate