This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author tim.peters
Recipients jneb, tim.peters
Date 2013-12-16.17:25:36
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1387214737.38.0.685132557079.issue19993@psf.upfronthosting.co.za>
In-reply-to
Content
Nice to see you, Jurjen!  Been a long time :-)

I'd like to see changes here too.  It's unclear what "a lazy version"  is intended to mean, exactly, but I agree the actual behavior is surprising, and that mpool.py is a lot less surprising in several ways.

I got bitten by this just last week, when running a parallelized search over a massive space _expected_ to succeed after exploring a tiny fraction of the search space.  Ran out of system resources because imap_unordered() tried to queue up countless millions of work descriptions.  I had hoped/expected that it would interleave generating and queue'ing "a few" inputs with retrieving outputs, much as mpool.py behaves.

In that case I switched to using apply_async() instead, interposing my own bounded queue (a collections.deque used only in the main program) to throttle the main program.  I'm still surprised it was necessary ;-)
History
Date User Action Args
2013-12-16 17:25:37tim.peterssetrecipients: + tim.peters, jneb
2013-12-16 17:25:37tim.peterssetmessageid: <1387214737.38.0.685132557079.issue19993@psf.upfronthosting.co.za>
2013-12-16 17:25:37tim.peterslinkissue19993 messages
2013-12-16 17:25:36tim.peterscreate