This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: multiprocessing.pool.Pool.map should take more than one iterable
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: davin Nosy List: Ido Michael, Jason Yu, aeros, berker.peksag, davin, flavianhautbois, miss-islington, naught101, rhettinger, tim.peters
Priority: normal Keywords: easy, patch

Created on 2016-08-26 23:15 by Jason Yu, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
mp.map.starmap.patch naught101, 2016-09-28 02:49 review
Pull Requests
URL Status Linked Edit
PR 17367 closed asaka, 2019-11-24 14:05
PR 17436 merged asaka, 2019-12-02 15:03
PR 17458 merged miss-islington, 2019-12-03 23:31
PR 17459 merged miss-islington, 2019-12-03 23:31
Messages (18)
msg273741 - (view) Author: Jason Yu (Jason Yu) Date: 2016-08-26 23:15
Hello all, 

The official doc describes `Pool.map(func, iterable[, chunksize])` as "A parallel equivalent of the map() built-in function.". 

Since the function signature of `map` is `map(function, iterable, ...)`, I hereby suggest that `Pool.map` should change its function signature to `Pool.map(function, iterable, ... [, chunksize])`. This will bring true equivalent to these functions. 

Tell me what you think. 

Pool.map: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map 

map: https://docs.python.org/3/library/functions.html#map 

Jason
msg273742 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-27 00:48
-0 We need better motivation than "that is was map() does".  Instead, are should look at whether there are compelling use cases and its impact on the complexity of the multiprocessing implementation.  I usually just prezip my data and it works fine.
msg273743 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2016-08-27 00:59
Note that `Pool` grew `starmap()` and `starmap_async()` methods in Python 3.3 to (mostly) address this.

The signature difference from the old builtin `map()` remains regrettable.  Note that the `Pool` version differs from the `concurrent.futures` version of `map()` in this respect too.
msg277478 - (view) Author: (naught101) Date: 2016-09-27 00:04
It would be helpful if the docs at https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map and in `help(Pool.map)` mentioned starmap in a "see also" section at the bottom.
msg277535 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-27 18:19
I think we can add a note for starmap() in the following sentence:

> [...] (it supports only one iterable argument though).

(Quoted from https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool.map)

Would you like to write a patch?
msg277572 - (view) Author: (naught101) Date: 2016-09-28 02:22
Happy to. What is the process? hg clone cpython, and make a patch from that and post it here? Or is there some pull-request style process available?
msg277573 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-09-28 02:30
> What is the process? hg clone cpython, and make a patch from that and post it here?

Yes, it's exactly what you just described :) See https://docs.python.org/devguide/index.html for details.

> Or is there some pull-request style process available?

Not yet, but we will move to GitHub soon.
msg277574 - (view) Author: (naught101) Date: 2016-09-28 02:35
OK, one question, is Pool.map preferable to Pool.starmap in any particular cases?
msg277576 - (view) Author: (naught101) Date: 2016-09-28 02:49
Here you go.
msg277680 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-29 02:21
@naught101: Thanks for the wording change -- I think your suggested doc change will be a good one.  I've added a comment which you can access via the "review" link to the right of the link for your mp.map.starmap.patch attachment.
msg348177 - (view) Author: Flavian Hautbois (flavianhautbois) * Date: 2019-07-19 13:56
Are you going to open a PR for this patch? I think that would be a nice addition
msg352973 - (view) Author: Ido Michael (Ido Michael) * Date: 2019-09-22 12:09
Hey,

Is it still open?
What else needs to be done?

Ido
msg355203 - (view) Author: Kyle Stanley (aeros) * (Python committer) Date: 2019-10-23 01:33
> Is it still open? What else needs to be done?

Yes, this patch needs to be translated into a GitHub PR. See https://devguide.python.org/pullrequest/ for more information on our PR workflow if you're not already familiar with it. Since naught101 wrote a patch, we should have their name in "Co-Authored-By" of the final commit to the PR if they have a GitHub account. If not, just mention it in a comment to the PR that they originally wrote the patch. 

Feel free to @mention me when you open the PR so I can help to review it, although this will likely require a final approval from a maintainer of multiprocessing module before it can be merged.
msg357769 - (view) Author: miss-islington (miss-islington) Date: 2019-12-03 23:31
New changeset eb48a451e3844185b9a8751c9badffbddc89689d by Miss Islington (bot) (An Long) in branch 'master':
bpo-27873: Update docstring for multiprocessing.Pool.map (GH-17436)
https://github.com/python/cpython/commit/eb48a451e3844185b9a8751c9badffbddc89689d
msg357770 - (view) Author: miss-islington (miss-islington) Date: 2019-12-03 23:36
New changeset 55a7046471e19843a68d4a1a15252fd197bb6913 by Miss Islington (bot) in branch '3.7':
bpo-27873: Update docstring for multiprocessing.Pool.map (GH-17436)
https://github.com/python/cpython/commit/55a7046471e19843a68d4a1a15252fd197bb6913
msg357771 - (view) Author: miss-islington (miss-islington) Date: 2019-12-03 23:37
New changeset baf07395eaa77e515ddfa1d3f42785d50b4d2889 by Miss Islington (bot) in branch '3.8':
bpo-27873: Update docstring for multiprocessing.Pool.map (GH-17436)
https://github.com/python/cpython/commit/baf07395eaa77e515ddfa1d3f42785d50b4d2889
msg362085 - (view) Author: Ido Michael (Ido Michael) * Date: 2020-02-16 17:22
Can this issue be closed, I see it was merged successfully?
msg362112 - (view) Author: Kyle Stanley (aeros) * (Python committer) Date: 2020-02-16 23:15
> Can this issue be closed, I see it was merged successfully?

Yep, I'll go ahead and close the issue now. Thanks for the reminder; I got caught up in a few other projects.
History
Date User Action Args
2022-04-11 14:58:35adminsetgithub: 72060
2020-02-16 23:15:41aerossetstatus: open -> closed

messages: + msg362112
stage: patch review -> resolved
2020-02-16 17:22:25Ido Michaelsetmessages: + msg362085
2019-12-03 23:37:44miss-islingtonsetmessages: + msg357771
2019-12-03 23:36:51miss-islingtonsetmessages: + msg357770
2019-12-03 23:31:15miss-islingtonsetpull_requests: + pull_request16940
2019-12-03 23:31:08miss-islingtonsetpull_requests: + pull_request16939
2019-12-03 23:31:07miss-islingtonsetnosy: + miss-islington
messages: + msg357769
2019-12-02 15:03:09asakasetpull_requests: + pull_request16916
2019-11-24 14:05:36asakasetstage: needs patch -> patch review
pull_requests: + pull_request16853
2019-10-23 01:33:34aerossetnosy: + aeros
messages: + msg355203
2019-09-22 12:09:43Ido Michaelsetnosy: + Ido Michael
messages: + msg352973
2019-07-19 13:56:00flavianhautboissetnosy: + flavianhautbois
messages: + msg348177
2016-09-29 02:21:24davinsetmessages: + msg277680
2016-09-28 02:49:26naught101setfiles: + mp.map.starmap.patch
keywords: + patch
messages: + msg277576
2016-09-28 02:35:18naught101setmessages: + msg277574
2016-09-28 02:30:15berker.peksagsetmessages: + msg277573
2016-09-28 02:22:30naught101setmessages: + msg277572
2016-09-27 18:19:41berker.peksagsetcomponents: + Documentation, - Interpreter Core
versions: + Python 3.5, Python 3.7
keywords: + easy
nosy: + berker.peksag

messages: + msg277535
stage: needs patch
2016-09-27 00:04:21naught101setnosy: + naught101
messages: + msg277478
2016-08-27 00:59:28tim.peterssetnosy: + tim.peters
messages: + msg273743
2016-08-27 00:48:26rhettingersetassignee: davin

messages: + msg273742
nosy: + rhettinger, davin
2016-08-26 23:15:50Jason Yucreate