Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing Pool not working for userdefined function #68390

Closed
abheeman mannequin opened this issue May 15, 2015 · 4 comments
Closed

Multiprocessing Pool not working for userdefined function #68390

abheeman mannequin opened this issue May 15, 2015 · 4 comments
Labels
OS-windows type-bug An unexpected behavior, bug, or error

Comments

@abheeman
Copy link
Mannequin

abheeman mannequin commented May 15, 2015

BPO 24202
Nosy @pfmoore, @tjguk, @zware, @zooba, @applio

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-15.14:07:07.520>
created_at = <Date 2015-05-15.12:45:24.738>
labels = ['type-bug', 'invalid', 'OS-windows']
title = 'Multiprocessing Pool not working for userdefined function'
updated_at = <Date 2015-05-15.14:07:07.519>
user = 'https://bugs.python.org/abheeman'

bugs.python.org fields:

activity = <Date 2015-05-15.14:07:07.519>
actor = 'paul.moore'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-15.14:07:07.520>
closer = 'paul.moore'
components = ['Windows']
creation = <Date 2015-05-15.12:45:24.738>
creator = 'abheeman'
dependencies = []
files = []
hgrepos = []
issue_num = 24202
keywords = []
message_count = 4.0
messages = ['243266', '243267', '243268', '243269']
nosy_count = 6.0
nosy_names = ['paul.moore', 'tim.golden', 'zach.ware', 'steve.dower', 'davin', 'abheeman']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24202'
versions = ['Python 2.7']

@abheeman
Copy link
Mannequin Author

abheeman mannequin commented May 15, 2015

I was looking to implement multiprocess pool. It worked fine with the numpy function while with the user defined function it ran into error.

import numpy
>>> import multiprocessing
>>> P = multiprocessing.Pool(5)
>>> P.map(numpy.sqrt,range(50))
 [0.0, 1.0, 1.4142135623730951, 1.7320508075688772, 2.0, 2.2360679774997898,     2.4494897427831779, 2.6457513110645907, 2.8284271247461903, 3.0, 3.1622776601683795, 3.3166247903553998, 3.4641016151377544, 3.6055512754639891, 3.7416573867739413, 3.872983346207417, 4.0, 4.1231056256176606, 4.2426406871192848, 4.358898943540674, 4.4721359549995796, 4.5825756949558398, 4.6904157598234297, 4.7958315233127191, 4.8989794855663558, 5.0, 5.0990195135927845, 5.196152422706632, 5.2915026221291814, 5.3851648071345037, 5.4772255750516612, 5.5677643628300215, 5.6568542494923806, 5.7445626465380286, 5.8309518948453007, 5.9160797830996161, 6.0, 6.0827625302982193, 6.164414002968976, 6.2449979983983983, 6.324555320336759, 6.4031242374328485, 6.4807406984078604, 6.5574385243020004, 6.6332495807107996, 6.7082039324993694, 6.7823299831252681, 6.8556546004010439, 6.9282032302755088, 7.0]
>>> def f(x):
      return x*x
>>> P.map(f, range(50))
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python27\lib\threading.py", line 530, in __bootstrap_inner
self.run()
File "C:\Python27\lib\threading.py", line 483, in run
self.__target(*self.__args, **self.__kwargs)
File "C:\Python27\lib\multiprocessing\pool.py", line 285, in _handle_tasks
put(task)
TypeError: expected string or Unicode object, NoneType found

@abheeman abheeman mannequin added type-crash A hard crash of the interpreter, possibly with a core dump OS-windows labels May 15, 2015
@pfmoore
Copy link
Member

pfmoore commented May 15, 2015

Multiprocessing works by firing up additional processes. Those processes won't have access to functions defined in the interactive interpreter.

Can you reproduce this problem in a standalone script? I suspect not, but if you can please post the script here.

Marking as "not a bug", but I haven't closed it yet in case a script reproducing the issue can be provided.

@applio applio added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels May 15, 2015
@abheeman
Copy link
Mannequin Author

abheeman mannequin commented May 15, 2015

Yes actually it produce no erroe on standalone script. But the script
executes without any outputs. not even for numpy function. I used code
below in script.
import multiprocessing
import numpy
def f(x):
return x*x

if __name__ = "__main__":
    p= multiprocessing.Pool(5)
    print p.map(numpy.sqrt,[1,2,3,4])
    print p.map(f,[1,2,3,4])
 On May 15, 2015 2:52 PM, "Paul Moore" <report@bugs.python.org> wrote:

Paul Moore added the comment:

Multiprocessing works by firing up additional processes. Those processes
won't have access to functions defined in the interactive interpreter.

Can you reproduce this problem in a standalone script? I suspect not, but
if you can please post the script here.

Marking as "not a bug", but I haven't closed it yet in case a script
reproducing the issue can be provided.

----------
nosy: +paul.moore
resolution: -> not a bug


Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue24202\>


@pfmoore
Copy link
Member

pfmoore commented May 15, 2015

OK, if it's not reproducible in a standalone script, I'll close this as it's expected behaviour.

Correcting the typo in your script (name == "__main__"), I ran it and it worked as expected on my system:

type multi.py
import multiprocessing
import numpy
def f(x):
return x*x

if __name__ == "__main__":
    p= multiprocessing.Pool(5)
    print(p.map(numpy.sqrt,[1,2,3,4]))
    print(p.map(f,[1,2,3,4]))
PS 15:05 {00:00.089} C:\Work\Scratch
>py .\multi.py
[1.0, 1.4142135623730951, 1.7320508075688772, 2.0]
[1, 4, 9, 16]

I'm not sure why you weren't getting output, but it doesn't look like a Python issue.

@pfmoore pfmoore closed this as completed May 15, 2015
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OS-windows type-bug An unexpected behavior, bug, or error
Projects
None yet
Development

No branches or pull requests

2 participants