classification
Title: multiprocessing docs bury very important 'callback=' args
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Mariatta, chadmiller-amzn, davin, docs@python, mdk, ned.deily, pitrou
Priority: normal Keywords: patch

Created on 2018-03-15 18:50 by chadmiller-amzn, last changed 2019-09-13 17:16 by davin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6122 closed chadmiller-amzn, 2018-03-15 21:10
PR 6455 closed chadmiller-amzn, 2018-04-12 01:05
Messages (9)
msg313905 - (view) Author: Chad (chadmiller-amzn) * Date: 2018-03-15 18:50
Callbacks are really important in multiprocessing. Doc writer almost ignores them.
msg313906 - (view) Author: Chad (chadmiller-amzn) * Date: 2018-03-15 18:55
https://github.com/chadmiller-amzn/cpython/pull/1

(Putting that in "GitHub PR" field says "Edit Error: Unknown PR format, acceptable formats are: full github URL, #pr_number, pr_number")
msg313915 - (view) Author: Mariatta (Mariatta) * (Python committer) Date: 2018-03-15 20:46
Can you make your PR in python/cpython GitHub repo, instead of your own fork?
msg314162 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-03-20 19:45
I don't really agree that "callbacks are really important in multiprocessing" (I think I've hardly ever used them).  I agree with the other doc changes in the PR.
msg314267 - (view) Author: Chad (chadmiller-amzn) * Date: 2018-03-22 15:52
On topic: My CLA is signed as of Monday, 19 March. My status here is not updated yet.


pitrou, off-topic: Without callbacks, users who want multiprocessing functions to return something, not just mutate state somewhere else, must gather jobs in a list and continually iterate through them polling to see if each has finished yet and conditionally popping it from the list. It's expensive and ugly and error-prone. Callbacks are really great, you should try them again.

So much better:

pool.apply_async(func, args, callback=when_finished_call_with_result)
msg319100 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-06-08 19:47
These doc changes have been languishing.  We have a difference of opinion about adding a callback example.  I don't have a strong feeling one way or the other.  Davin or anyone else, what say you?
msg352366 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2019-09-13 16:55
I agree Antoine on this one, if one want the result, It'll get it from the  returned value (.get method in the example), or simply by using the not-async versions and directly get the results as a return value of the call.

Also the given example in the documentation won't work as the result is never waited for, the pool gets destroyed before the sleep have the time to sleep.
msg352367 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2019-09-13 16:55
I'd wait for Davin's review, but I'd keep the documentation as they are.
msg352375 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-09-13 17:16
I appreciate the functionality offered by the callbacks and have found good uses for them, as Chad clearly does/has.  That said, the thought of expanding the documentation on the callbacks had not come up for me.

Reading through the proposed changes to the prose explanations, the choice of words has changed but not significantly and virtually no new concepts are being explained.

I agree with Julien that the docs should stay as they are.

Chad:  Thank you for advocating for things you think more people need to know about even if we do not update the docs this time.
History
Date User Action Args
2019-09-13 17:16:38davinsetstatus: open -> closed
resolution: rejected
messages: + msg352375

stage: patch review -> resolved
2019-09-13 16:55:59mdksetmessages: + msg352367
2019-09-13 16:55:25mdksetnosy: + mdk
messages: + msg352366
2018-06-08 19:47:47ned.deilysetnosy: + ned.deily
messages: + msg319100
2018-04-12 01:05:48chadmiller-amznsetpull_requests: + pull_request6150
2018-03-22 15:52:55chadmiller-amznsetmessages: + msg314267
2018-03-20 19:45:29pitrousetmessages: + msg314162
2018-03-15 21:10:27chadmiller-amznsetkeywords: + patch
stage: patch review
pull_requests: + pull_request5886
2018-03-15 20:46:42Mariattasetversions: - Python 3.4, Python 3.5
2018-03-15 20:46:10Mariattasetnosy: + Mariatta
messages: + msg313915
2018-03-15 18:55:08chadmiller-amznsetmessages: + msg313906
versions: + Python 3.4, Python 3.5
2018-03-15 18:51:31ned.deilysetnosy: + pitrou, davin

versions: - Python 3.4, Python 3.5
2018-03-15 18:50:50chadmiller-amzncreate