classification
Title: multiprocessing.pool.Pool.map should take more than one iterable
Type: enhancement Stage: needs patch
Components: Documentation Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: davin Nosy List: Jason Yu, berker.peksag, davin, naught101, rhettinger, tim.peters
Priority: normal Keywords: easy, patch

Created on 2016-08-26 23:15 by Jason Yu, last changed 2016-09-29 02:21 by davin.

Files
File name Uploaded Description Edit
mp.map.starmap.patch naught101, 2016-09-28 02:49 review
Messages (10)
msg273741 - (view) Author: Jason Yu (Jason Yu) Date: 2016-08-26 23:15
Hello all, 

The official doc describes `Pool.map(func, iterable[, chunksize])` as "A parallel equivalent of the map() built-in function.". 

Since the function signature of `map` is `map(function, iterable, ...)`, I hereby suggest that `Pool.map` should change its function signature to `Pool.map(function, iterable, ... [, chunksize])`. This will bring true equivalent to these functions. 

Tell me what you think. 

Pool.map: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map 

map: https://docs.python.org/3/library/functions.html#map 

Jason
msg273742 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-27 00:48
-0 We need better motivation than "that is was map() does".  Instead, are should look at whether there are compelling use cases and its impact on the complexity of the multiprocessing implementation.  I usually just prezip my data and it works fine.
msg273743 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2016-08-27 00:59
Note that `Pool` grew `starmap()` and `starmap_async()` methods in Python 3.3 to (mostly) address this.

The signature difference from the old builtin `map()` remains regrettable.  Note that the `Pool` version differs from the `concurrent.futures` version of `map()` in this respect too.
msg277478 - (view) Author: (naught101) Date: 2016-09-27 00:04
It would be helpful if the docs at https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map and in `help(Pool.map)` mentioned starmap in a "see also" section at the bottom.
msg277535 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-27 18:19
I think we can add a note for starmap() in the following sentence:

> [...] (it supports only one iterable argument though).

(Quoted from https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map)

Would you like to write a patch?
msg277572 - (view) Author: (naught101) Date: 2016-09-28 02:22
Happy to. What is the process? hg clone cpython, and make a patch from that and post it here? Or is there some pull-request style process available?
msg277573 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-28 02:30
> What is the process? hg clone cpython, and make a patch from that and post it here?

Yes, it's exactly what you just described :) See https://docs.python.org/devguide/index.html for details.

> Or is there some pull-request style process available?

Not yet, but we will move to GitHub soon.
msg277574 - (view) Author: (naught101) Date: 2016-09-28 02:35
OK, one question, is Pool.map preferable to Pool.starmap in any particular cases?
msg277576 - (view) Author: (naught101) Date: 2016-09-28 02:49
Here you go.
msg277680 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-29 02:21
@naught101: Thanks for the wording change -- I think your suggested doc change will be a good one.  I've added a comment which you can access via the "review" link to the right of the link for your mp.map.starmap.patch attachment.
History
Date User Action Args
2016-09-29 02:21:24davinsetmessages: + msg277680
2016-09-28 02:49:26naught101setfiles: + mp.map.starmap.patch
keywords: + patch
messages: + msg277576
2016-09-28 02:35:18naught101setmessages: + msg277574
2016-09-28 02:30:15berker.peksagsetmessages: + msg277573
2016-09-28 02:22:30naught101setmessages: + msg277572
2016-09-27 18:19:41berker.peksagsetcomponents: + Documentation, - Interpreter Core
versions: + Python 3.5, Python 3.7
keywords: + easy
nosy: + berker.peksag

messages: + msg277535
stage: needs patch
2016-09-27 00:04:21naught101setnosy: + naught101
messages: + msg277478
2016-08-27 00:59:28tim.peterssetnosy: + tim.peters
messages: + msg273743
2016-08-27 00:48:26rhettingersetassignee: davin

messages: + msg273742
nosy: + rhettinger, davin
2016-08-26 23:15:50Jason Yucreate