New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Multiprocessing Pool not working for userdefined function #68390
Comments
I was looking to implement multiprocess pool. It worked fine with the numpy function while with the user defined function it ran into error. import numpy
>>> import multiprocessing
>>> P = multiprocessing.Pool(5)
>>> P.map(numpy.sqrt,range(50))
[0.0, 1.0, 1.4142135623730951, 1.7320508075688772, 2.0, 2.2360679774997898, 2.4494897427831779, 2.6457513110645907, 2.8284271247461903, 3.0, 3.1622776601683795, 3.3166247903553998, 3.4641016151377544, 3.6055512754639891, 3.7416573867739413, 3.872983346207417, 4.0, 4.1231056256176606, 4.2426406871192848, 4.358898943540674, 4.4721359549995796, 4.5825756949558398, 4.6904157598234297, 4.7958315233127191, 4.8989794855663558, 5.0, 5.0990195135927845, 5.196152422706632, 5.2915026221291814, 5.3851648071345037, 5.4772255750516612, 5.5677643628300215, 5.6568542494923806, 5.7445626465380286, 5.8309518948453007, 5.9160797830996161, 6.0, 6.0827625302982193, 6.164414002968976, 6.2449979983983983, 6.324555320336759, 6.4031242374328485, 6.4807406984078604, 6.5574385243020004, 6.6332495807107996, 6.7082039324993694, 6.7823299831252681, 6.8556546004010439, 6.9282032302755088, 7.0]
>>> def f(x):
return x*x >>> P.map(f, range(50))
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 530, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Python27\lib\multiprocessing\pool.py", line 285, in _handle_tasks
put(task)
TypeError: expected string or Unicode object, NoneType found |
Multiprocessing works by firing up additional processes. Those processes won't have access to functions defined in the interactive interpreter. Can you reproduce this problem in a standalone script? I suspect not, but if you can please post the script here. Marking as "not a bug", but I haven't closed it yet in case a script reproducing the issue can be provided. |
Yes actually it produce no erroe on standalone script. But the script if __name__ = "__main__":
p= multiprocessing.Pool(5)
print p.map(numpy.sqrt,[1,2,3,4])
print p.map(f,[1,2,3,4])
On May 15, 2015 2:52 PM, "Paul Moore" <report@bugs.python.org> wrote:
|
OK, if it's not reproducible in a standalone script, I'll close this as it's expected behaviour. Correcting the typo in your script (name == "__main__"), I ran it and it worked as expected on my system:
if __name__ == "__main__":
p= multiprocessing.Pool(5)
print(p.map(numpy.sqrt,[1,2,3,4]))
print(p.map(f,[1,2,3,4]))
PS 15:05 {00:00.089} C:\Work\Scratch
>py .\multi.py
[1.0, 1.4142135623730951, 1.7320508075688772, 2.0]
[1, 4, 9, 16] I'm not sure why you weren't getting output, but it doesn't look like a Python issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: