Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiprocessing not compatible with functools.partial #49478

Closed
ndbecker mannequin opened this issue Feb 12, 2009 · 11 comments
Closed

multiprocessing not compatible with functools.partial #49478

ndbecker mannequin opened this issue Feb 12, 2009 · 11 comments
Assignees
@jackdied
Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@ndbecker
Copy link
Mannequin

ndbecker mannequin commented Feb 12, 2009

BPO 5228
Nosy @amauryfa, @pitrou, @jackdied, @tiran, @jwilk

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = 'https://github.com/jackdied'
closed_at = <Date 2009-03-31.23:48:06.735>
created_at = <Date 2009-02-12.14:26:28.378>
labels = ['type-bug', 'library']
title = 'multiprocessing not compatible with functools.partial'
updated_at = <Date 2011-04-03.21:39:03.325>
user = 'https://bugs.python.org/ndbecker'

bugs.python.org fields:

activity = <Date 2011-04-03.21:39:03.325>
actor = 'jwilk'
assignee = 'jackdied'
closed = True
closed_date = <Date 2009-03-31.23:48:06.735>
closer = 'jackdied'
components = ['Library (Lib)']
creation = <Date 2009-02-12.14:26:28.378>
creator = 'ndbecker'
dependencies = []
files = []
hgrepos = []
issue_num = 5228
keywords = []
message_count = 11.0
messages = ['81766', '81767', '81788', '81789', '84871', '84872', '84949', '100669', '130511', '130513', '130548']
nosy_count = 9.0
nosy_names = ['amaury.forgeotdarc', 'pitrou', 'jackdied', 'christian.heimes', 'jwilk', 'jnoller', 'ndbecker', 'joseph.h.garvin', 'uptimebox']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5228'
versions = ['Python 2.6']
@ndbecker
Copy link
Mannequin Author

ndbecker mannequin commented Feb 12, 2009 

from multiprocessing import Pool

def power (x, pwr=2):
    return x**pwr

import functools
run_test = functools.partial (power, pwr=3)

if __name__ == "__main__":

    pool = Pool()
    cases = [3,4,5]
    results = pool.map (run_test, cases)
TypeError: type 'partial' takes at least one argument
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self.run()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    task = get()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    return recv()
TypeError: type 'partial' takes at least one argument

@ndbecker ndbecker mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Feb 12, 2009
@jnoller jnoller mannequin self-assigned this Feb 12, 2009
@jnoller
Copy link
Mannequin

jnoller mannequin commented Feb 12, 2009

See this mail thread:
http://mail.python.org/pipermail/python-dev/2009-February/086034.html

@pitrou
Copy link
Member

pitrou commented Feb 12, 2009

Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.

@jnoller
Copy link
Mannequin

jnoller mannequin commented Feb 12, 2009

Antoine Pitrou <pitrou@free.fr> added the comment:

Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.

I would tend to agree

@jnoller
Copy link
Mannequin

jnoller mannequin commented Mar 31, 2009

I agree that this is a nice feature, however it requires adding a
getstate/setstate within the functools C code.

I would need a patch which does this, and adds tests to the
functools/pickle test suite for this. Also, the feature will only be
accepted for py3k (3.1) currently. Features such as this will not be
backported to the 2.x branch

@jnoller
Copy link
Mannequin

jnoller mannequin commented Mar 31, 2009

Jack offered to take a peek

@jnoller jnoller mannequin assigned jackdied and unassigned jnoller Mar 31, 2009
@jackdied
Copy link
Contributor

Fixed rev 70931. Happy pickling!

@josephhgarvin
Copy link
Mannequin

josephhgarvin mannequin commented Mar 8, 2010

I think this bug still exists in Python 2.6.4, and I haven't tested 2.6.5, but since 2.6.4 was released 6 months after this bug was closed I assume it's still an issue. Running ndbecker's test program as is produces the following output on Solaris 10:

Process PoolWorker-1:
Process PoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process PoolWorker-3:
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    self.run()
    task = get()
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
TypeError: type 'partial' takes at least one argument

@uptimebox
Copy link
Mannequin

uptimebox mannequin commented Mar 10, 2011

The bug still exists in 2.6.6

Process PoolWorker-2:
Traceback (most recent call last):
Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
    task = get()
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument

@amauryfa
Copy link
Member

The bug was fixed in 2.7, and not backported to 2.6.
2.6 is now in security-fix-only mode, if possible you should upgrade to 2.7.

@uptimebox
Copy link
Mannequin

uptimebox mannequin commented Mar 11, 2011

Unfortunately, upgrading to 2.7 is not an option for me, since my application works on Debian stable which only includes 2.6. For now I'm settled with the following workaround: http://paste.pocoo.org/show/351774/

@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jackdied jackdied

Labels
stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jackdied @amauryfa @pitrou