Message308728
Yes, sorry for not being quite clear the first time around :)
I eventually found out about Pool.imap (see item 3 on list in OP) and indeed it fits my use case very nicely, but my point was that the documentation is somewhat misleading with respect to the semantics of built-in `map()` in Python 3.
Specifically, I would argue that it is unexpected for a function which claims to be "Equivalent to map(func, *iterables)" to require allocating a list the length of the shortest iterable.
Maybe a code example will make this clearer for potential newcomers to the discussion -- this is what I would expect to happen (= the behavior of built-in `map()` itself), yielding values from the iterable is interleaved with calls to the mapped function:
```
>>> def gen():
... for i in range(3):
... print("yielding", i)
... yield i
...
>>> def add1(i):
... print("adding 1 to", i)
... return i + 1
...
>>> list(map(add1, gen()))
yielding 0
adding 1 to 0
yielding 1
adding 1 to 1
yielding 2
adding 1 to 2
[1, 2, 3]
```
This is what happens instead with `concurrent.futures.Executor.map()`:
```
>>> def my_map(fn, iterable):
... lst = list(iterable)
... for i in lst:
... yield fn(i)
...
>>> list(my_map(add1, gen()))
yielding 0
yielding 1
yielding 2
adding 1 to 0
adding 1 to 1
adding 1 to 2
[1, 2, 3]
``` |
|
Date |
User |
Action |
Args |
2017-12-20 12:06:40 | dlukes | set | recipients:
+ dlukes, pitrou, docs@python |
2017-12-20 12:06:40 | dlukes | set | messageid: <1513771600.8.0.213398074469.issue32306@psf.upfronthosting.co.za> |
2017-12-20 12:06:40 | dlukes | link | issue32306 messages |
2017-12-20 12:06:40 | dlukes | create | |
|