Author josh.r
Recipients josh.r
Date 2017-03-18.01:32:54
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1489800776.56.0.575396558681.issue29842@psf.upfronthosting.co.za>
In-reply-to
Content
As currently implemented, Executor.map is not particularly lazy. Specifically, if given huge argument iterables, it will not begin yielding results until all tasks have been submitted; if given an infinite input iterable, it will run out of memory before yielding a single result.

This makes it unusable as a drop in replacement for plain map, which, being lazy, handles infinite iterables just fine, and produces results promptly.

Proposed change makes Executor.map begin yielding results for large iterables without submitting every task up front. As a reasonable default, I have it submit a number of tasks equal to twice the number of workers, submitting a new task immediately after getting results for the next future in line, before yielding the result (to ensure the number of outstanding futures stays constant). A new keyword-only argument, prefetch, is provided to explicitly specify how many tasks should be queued above and beyond the number of workers.

Working on submitting pull request now.
History
Date User Action Args
2017-03-18 01:32:56josh.rsetrecipients: + josh.r
2017-03-18 01:32:56josh.rsetmessageid: <1489800776.56.0.575396558681.issue29842@psf.upfronthosting.co.za>
2017-03-18 01:32:56josh.rlinkissue29842 messages
2017-03-18 01:32:54josh.rcreate