classification
Title: Make Executor.map work with infinite/large inputs correctly
Type: Stage:
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: bquinlan, ezio.melotti, josh.r
Priority: normal Keywords:

Created on 2017-03-18 01:32 by josh.r, last changed 2017-03-19 13:19 by josh.r.

Pull Requests
URL Status Linked Edit
PR 707 open python-dev, 2017-03-18 02:13
Messages (2)
msg289789 - (view) Author: Josh Rosenberg (josh.r) * Date: 2017-03-18 01:32
As currently implemented, Executor.map is not particularly lazy. Specifically, if given huge argument iterables, it will not begin yielding results until all tasks have been submitted; if given an infinite input iterable, it will run out of memory before yielding a single result.

This makes it unusable as a drop in replacement for plain map, which, being lazy, handles infinite iterables just fine, and produces results promptly.

Proposed change makes Executor.map begin yielding results for large iterables without submitting every task up front. As a reasonable default, I have it submit a number of tasks equal to twice the number of workers, submitting a new task immediately after getting results for the next future in line, before yielding the result (to ensure the number of outstanding futures stays constant). A new keyword-only argument, prefetch, is provided to explicitly specify how many tasks should be queued above and beyond the number of workers.

Working on submitting pull request now.
msg289790 - (view) Author: Josh Rosenberg (josh.r) * Date: 2017-03-18 02:18
Nosying folks suggested by GitHub, hope that's the right etiquette.

For the record, filled out contributor agreement ages ago, but hadn't linked (or even created) GitHub account until after I got the warning. I've linked this account to my GitHub username now, hope that's sufficient.
History
Date User Action Args
2017-03-19 13:19:29josh.rsettitle: Executor.map should not submit all futures prior to yielding any results -> Make Executor.map work with infinite/large inputs correctly
2017-03-18 02:18:42josh.rsetnosy: + bquinlan, ezio.melotti
messages: + msg289790
2017-03-18 02:13:29python-devsetpull_requests: + pull_request627
2017-03-18 01:32:56josh.rcreate