This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Multiprocessing Pool not working for userdefined function
Type: behavior Stage:
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: abheeman, davin, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2015-05-15 12:45 by abheeman, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (4)
msg243266 - (view) Author: Abhishek Manandhar (abheeman) Date: 2015-05-15 12:45
I was looking to implement multiprocess pool. It worked fine with the numpy function while with the user defined function it ran into error.

import numpy
>>> import multiprocessing
>>> P = multiprocessing.Pool(5)
>>> P.map(numpy.sqrt,range(50))
 [0.0, 1.0, 1.4142135623730951, 1.7320508075688772, 2.0, 2.2360679774997898,     2.4494897427831779, 2.6457513110645907, 2.8284271247461903, 3.0, 3.1622776601683795, 3.3166247903553998, 3.4641016151377544, 3.6055512754639891, 3.7416573867739413, 3.872983346207417, 4.0, 4.1231056256176606, 4.2426406871192848, 4.358898943540674, 4.4721359549995796, 4.5825756949558398, 4.6904157598234297, 4.7958315233127191, 4.8989794855663558, 5.0, 5.0990195135927845, 5.196152422706632, 5.2915026221291814, 5.3851648071345037, 5.4772255750516612, 5.5677643628300215, 5.6568542494923806, 5.7445626465380286, 5.8309518948453007, 5.9160797830996161, 6.0, 6.0827625302982193, 6.164414002968976, 6.2449979983983983, 6.324555320336759, 6.4031242374328485, 6.4807406984078604, 6.5574385243020004, 6.6332495807107996, 6.7082039324993694, 6.7823299831252681, 6.8556546004010439, 6.9282032302755088, 7.0]
>>> def f(x):
      return x*x

>>> P.map(f, range(50))
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 530, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Python27\lib\multiprocessing\pool.py", line 285, in _handle_tasks
put(task)
TypeError: expected string or Unicode object, NoneType found
msg243267 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-05-15 12:52
Multiprocessing works by firing up additional processes. Those processes won't have access to functions defined in the interactive interpreter.

Can you reproduce this problem in a standalone script? I suspect not, but if you can please post the script here.

Marking as "not a bug", but I haven't closed it yet in case a script reproducing the issue can be provided.
msg243268 - (view) Author: Abhishek Manandhar (abheeman) Date: 2015-05-15 13:56
Yes actually it produce no erroe on standalone script. But the script
executes without any outputs. not even for numpy function. I used code
below in script.
import multiprocessing
import numpy
def f(x):
    return x*x

if __name__ = "__main__":
    p= multiprocessing.Pool(5)
    print p.map(numpy.sqrt,[1,2,3,4])
    print p.map(f,[1,2,3,4])
 On May 15, 2015 2:52 PM, "Paul Moore" <report@bugs.python.org> wrote:

>
> Paul Moore added the comment:
>
> Multiprocessing works by firing up additional processes. Those processes
> won't have access to functions defined in the interactive interpreter.
>
> Can you reproduce this problem in a standalone script? I suspect not, but
> if you can please post the script here.
>
> Marking as "not a bug", but I haven't closed it yet in case a script
> reproducing the issue can be provided.
>
> ----------
> nosy: +paul.moore
> resolution:  -> not a bug
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue24202>
> _______________________________________
>
msg243269 - (view) Author: Paul Moore (paul.moore) * (Python committer) Date: 2015-05-15 14:07
OK, if it's not reproducible in a standalone script, I'll close this as it's expected behaviour.

Correcting the typo in your script (__name__ == "__main__"), I ran it and it worked as expected on my system:

>type multi.py
import multiprocessing
import numpy
def f(x):
    return x*x

if __name__ == "__main__":
    p= multiprocessing.Pool(5)
    print(p.map(numpy.sqrt,[1,2,3,4]))
    print(p.map(f,[1,2,3,4]))
PS 15:05 {00:00.089} C:\Work\Scratch
>py .\multi.py
[1.0, 1.4142135623730951, 1.7320508075688772, 2.0]
[1, 4, 9, 16]

I'm not sure why you weren't getting output, but it doesn't look like a Python issue.
History
Date User Action Args
2022-04-11 14:58:16adminsetgithub: 68390
2015-05-15 14:07:07paul.mooresetstatus: open -> closed

messages: + msg243269
2015-05-15 13:56:59abheemansetmessages: + msg243268
2015-05-15 13:03:14davinsetnosy: + davin
type: crash -> behavior
2015-05-15 12:52:08paul.mooresetresolution: not a bug

messages: + msg243267
nosy: + paul.moore
2015-05-15 12:45:24abheemancreate