Author dan.oreilly
Recipients bquinlan, dan.oreilly, pitrou, tbrink
Date 2014-07-24.01:59:05
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1406167147.73.0.0574943039345.issue11271@psf.upfronthosting.co.za>
In-reply-to
Content
I'm seeing an even larger difference between multiprocessing.Pool and ProcessPoolExecutor on my machine, with Python 3.4:

Starting multiproc...done in 2.160644769668579 s.
Starting futures...done in 67.953957319259644 s.
Starting futures "fixed"...done in 2.134932041168213 s.

I've updated the initial patch to address the comments Antoine made; the chunksize now defaults to 1, and itertools is used to chunk the input iterables, rather than building a list. Attached is an updated benchmark script:

Starting multiproc...done in 2.2295100688934326 s.
Starting futures...done in 68.5991039276123 s.
Starting futures "fixed" (no chunking)...done in 69.18992304801941 s.
Starting futures "fixed" (with chunking)...done in 2.352942705154419 s.

The new implementation of map has essentially identical performance to the original with chunksize=1, but it performs much better with a larger chunksize provided.
History
Date User Action Args
2014-07-24 01:59:07dan.oreillysetrecipients: + dan.oreilly, bquinlan, pitrou, tbrink
2014-07-24 01:59:07dan.oreillysetmessageid: <1406167147.73.0.0574943039345.issue11271@psf.upfronthosting.co.za>
2014-07-24 01:59:07dan.oreillylinkissue11271 messages
2014-07-24 01:59:07dan.oreillycreate