classification
Title: multiprocessing not compatible with functools.partial
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jackdied Nosy List: christian.heimes, jackdied, jnoller, ndbecker, pitrou (5)
Priority: low Keywords

Created on 2009-02-12 14:26 by ndbecker, last changed 2009-03-31 23:48 by jackdied.

Messages (7)
msg81766 - (view) Author: (ndbecker) Date: 2009-02-12 14:26
from multiprocessing import Pool

def power (x, pwr=2):
    return x**pwr

import functools
run_test = functools.partial (power, pwr=3)

if __name__ == "__main__":

    pool = Pool()
    cases = [3,4,5]
    results = pool.map (run_test, cases)

TypeError: type 'partial' takes at least one argument
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self.run()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    task = get()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    return recv()
TypeError: type 'partial' takes at least one argument
msg81767 - (view) Author: Jesse Noller (jnoller) Date: 2009-02-12 14:28
See this mail thread:
http://mail.python.org/pipermail/python-dev/2009-February/086034.html
msg81788 - (view) Author: Antoine Pitrou (pitrou) Date: 2009-02-12 18:01
Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.
msg81789 - (view) Author: Jesse Noller (jnoller) Date: 2009-02-12 18:03
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Rather than implement a multiprocessing-specific fix, it would certainly
> be better to make functools.partial picklable, as pointed out in the ML
> thread Jesse linked to.
>

I would tend to agree
msg84871 - (view) Author: Jesse Noller (jnoller) Date: 2009-03-31 19:13
I agree that this is a nice feature, however it requires adding a 
getstate/setstate within the functools C code. 

I would need a patch which does this, and adds tests to the 
functools/pickle test suite for this. Also, the feature will only be 
accepted for py3k (3.1) currently. Features such as this will not be 
backported to the 2.x branch
msg84872 - (view) Author: Jesse Noller (jnoller) Date: 2009-03-31 19:19
Jack offered to take a peek
msg84949 - (view) Author: Jack Diederich (jackdied) Date: 2009-03-31 23:48
Fixed rev 70931.  Happy pickling!
History
Date User Action Args
2009-03-31 23:48:06jackdiedsetstatus: open -> closed
resolution: fixed
messages: + msg84949
2009-03-31 19:19:12jnollersetassignee: jnoller -> jackdied

messages: + msg84872
nosy: + jackdied
2009-03-31 19:13:26jnollersetmessages: + msg84871
2009-03-29 14:37:58jnollersetpriority: low
2009-02-12 18:03:27jnollersetmessages: + msg81789
2009-02-12 18:01:51pitrousetnosy: + pitrou
messages: + msg81788
2009-02-12 14:28:26jnollersetmessages: + msg81767
2009-02-12 14:27:56jnollersetassignee: jnoller
nosy: + christian.heimes
2009-02-12 14:26:28ndbeckercreate