Title: Documentation: Clarify role of callback with map_async
Type: behavior Stage:
Components: Documentation Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Jose Miguel Colella, davin, docs@python
Priority: normal Keywords:

Created on 2016-12-29 21:57 by Jose Miguel Colella, last changed 2020-09-18 16:03 by iritkatriel.

File name Uploaded Description Edit Jose Miguel Colella, 2016-12-29 21:57
Messages (4)
msg284295 - (view) Author: Jose Miguel Colella (Jose Miguel Colella) Date: 2016-12-29 21:57
Hello I am trying to use the callback for the map_async method for Pool, but have found a bug. In the below code, only the print statement is carried out, the return is completely ignored. Is this working as designed or is this a bug?

from multiprocessing import Pool

def f(x):
    return x * x

def s(x):
    print(f'Here: {x}')
    return type(x)

if __name__ == '__main__':
    with Pool(5) as p:
        result = p.map_async(f, [1, 2, 3], callback=s)
        q = result.get()
msg284296 - (view) Author: Jose Miguel Colella (Jose Miguel Colella) Date: 2016-12-29 21:59
The result is:
Here: [1, 4, 9]
[1, 4, 9]
msg284305 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-12-30 03:20
This appears to be working as designed.  The purpose of the callback is to receive a single argument (nominally, the ready result from the map operation) and perform its task (whatever it may be) very quickly so as not to further delay the handing back of results from the map operation.

Attempting to read into your comments, it sounds like you were perhaps expecting the return from the callback to appear somewhere?  If that was your expectation then no, that is not the purpose of a callback here.  Any transformation of input data to the final output data is performed by the function being applied by the map.  While there are many potential uses for a callback, its stated requirement is that it be as quick an operation as possible as opposed to something that further transforms the results or anything like a "reduce" operation which is not necessarily quick.

If you think this needs clarification in the documentation, please suggest something?
msg284320 - (view) Author: Jose Miguel Colella (Jose Miguel Colella) Date: 2016-12-30 07:04
Hello David,
Thanks for your response. Improvements to the documentation could clear this misunderstanding. I had initially believed that after transforming with the function passed to the map, it would use the callback on each of the result arguments. 
Just to understand the use case of the callback. So basically it should not return anything and be a simple print?
Date User Action Args
2020-09-18 16:03:49iritkatrielsetassignee: docs@python

nosy: + docs@python
components: + Documentation, - Library (Lib)
title: Python 3.6.0 multiprocessing map_async callback -> Documentation: Clarify role of callback with map_async
2016-12-30 07:04:15Jose Miguel Colellasetmessages: + msg284320
2016-12-30 03:20:42davinsetmessages: + msg284305
2016-12-29 22:03:02ned.deilysetnosy: + davin
2016-12-29 21:59:26Jose Miguel Colellasetmessages: + msg284296
2016-12-29 21:57:31Jose Miguel Colellacreate