Author tbrink
Recipients tbrink
Date 2011-02-21.15:04:48
SpamBayes Score 6.48601e-10
Marked as misclassified No
Message-id <1298300689.71.0.822605356721.issue11271@psf.upfronthosting.co.za>
In-reply-to
Content
I tested the new concurrent.futures.ProcessPoolExecutor.map() in 3.2 with the is_prime() function from the documentation example. This was significantly slower than using multiprocessing.Pool.map(). Quick look at the source showed that multiprocessing sends the iterable in chunks to the worker process while futures sends always only one entry of the iterable to the worker.

Functions like is_prime() which finish relatively fast make the communication overhead (at least I guess that is the culprit) very big in comparison.

Attached is a file which demonstrates the problem and a quick workaround.  The workaround uses the chunk idea from multiprocessing.  The problem is that it requires the iterables passed to map() to have a length and be indexable with a slice.  I believe this limitation could be worked around.
History
Date User Action Args
2011-02-21 15:04:49tbrinksetrecipients: + tbrink
2011-02-21 15:04:49tbrinksetmessageid: <1298300689.71.0.822605356721.issue11271@psf.upfronthosting.co.za>
2011-02-21 15:04:48tbrinklinkissue11271 messages
2011-02-21 15:04:48tbrinkcreate