classification
Title: Support for iterators in multiprocessing map
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.3
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: Nosy List: acooke, pitrou, sbt
Priority: normal Keywords:

Created on 2011-09-05 12:00 by acooke, last changed 2012-06-11 17:03 by sbt. This issue is now closed.

Messages (6)
msg143511 - (view) Author: andrew cooke (acooke) Date: 2011-09-05 12:00
http://stackoverflow.com/questions/7306522/combining-itertools-and-multiprocessing/7307078 suggests (and the idea itself seems reasonable) that it would sometimes be useful for multiprocessing to operate correctly (ie lazily) with lazy input (iterables).  for example, if the input is large, or perhaps generated by some other process "on demand".

obviously this complicates matters, given the asynchronous nature of a worker pool, and would mean re-allocating the results list as required.  but in principle i suspect it would be possible and might be a useful extension.
msg143533 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-09-05 15:49
Since it's a feature request, I would suggest to look whether it can apply to concurrent.futures instead.
msg162523 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-08 11:59
If you want lazy operation then you should use imap(f, it[, chunksize]) rather than using map_async(f, it).

This will return an iterator rather than a list.  Also, the iterator's next() method has a timeout argument.  (chunksize is the number of items which get sent to a worker in each batch, with 1 being the default.)
msg162625 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-11 15:46
Unless you have a reason why imap() does not solve the problem I will eventually close the issue as rejected.
msg162626 - (view) Author: andrew cooke (acooke) Date: 2012-06-11 16:14
hi - i'm the original author (may be using a different account).  as far as i remember, i raised this because it seemed relevant given the link i gave.  if you've looked at the issue and think your approach would work, or that this should be closed, or whatever, that's fine by me.  i'm not going to check myself - i can't remember anything about this now (nearly a year later) and it's not my place to worry about your code (no offence - just trying to clarify that i have no skin in this game).
msg162629 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-06-11 17:03
I'll close then.
History
Date User Action Args
2012-06-11 17:03:51sbtsetstatus: open -> closed

messages: + msg162629
2012-06-11 16:14:13acookesetstatus: pending -> open

messages: + msg162626
2012-06-11 15:46:26sbtsetstatus: open -> pending
resolution: rejected
messages: + msg162625

stage: resolved
2012-06-08 11:59:10sbtsetnosy: + sbt
messages: + msg162523
2011-09-05 15:49:58pitrousetnosy: + pitrou

messages: + msg143533
versions: + Python 3.3, - Python 3.4
2011-09-05 12:00:45acookecreate