classification
Title: multiprocessing not compatible with functools.partial
Type: behavior Stage:
Components: Library (Lib) Versions: Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: jackdied Nosy List: amaury.forgeotdarc, christian.heimes, jackdied, jnoller, joseph.h.garvin, jwilk, ndbecker, pitrou, uptimebox
Priority: low Keywords:

Created on 2009-02-12 14:26 by ndbecker, last changed 2011-04-03 21:39 by jwilk. This issue is now closed.

Messages (11)
msg81766 - (view) Author: (ndbecker) Date: 2009-02-12 14:26
from multiprocessing import Pool

def power (x, pwr=2):
    return x**pwr

import functools
run_test = functools.partial (power, pwr=3)

if __name__ == "__main__":

    pool = Pool()
    cases = [3,4,5]
    results = pool.map (run_test, cases)

TypeError: type 'partial' takes at least one argument
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self.run()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    task = get()
  File "/usr/lib/python2.5/site-packages/multiprocessing-2.6.1.1-py2.5-
    return recv()
TypeError: type 'partial' takes at least one argument
msg81767 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-02-12 14:28
See this mail thread:
http://mail.python.org/pipermail/python-dev/2009-February/086034.html
msg81788 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-02-12 18:01
Rather than implement a multiprocessing-specific fix, it would certainly
be better to make functools.partial picklable, as pointed out in the ML
thread Jesse linked to.
msg81789 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-02-12 18:03
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> Rather than implement a multiprocessing-specific fix, it would certainly
> be better to make functools.partial picklable, as pointed out in the ML
> thread Jesse linked to.
>

I would tend to agree
msg84871 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-03-31 19:13
I agree that this is a nice feature, however it requires adding a 
getstate/setstate within the functools C code. 

I would need a patch which does this, and adds tests to the 
functools/pickle test suite for this. Also, the feature will only be 
accepted for py3k (3.1) currently. Features such as this will not be 
backported to the 2.x branch
msg84872 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-03-31 19:19
Jack offered to take a peek
msg84949 - (view) Author: Jack Diederich (jackdied) * (Python committer) Date: 2009-03-31 23:48
Fixed rev 70931.  Happy pickling!
msg100669 - (view) Author: (joseph.h.garvin) Date: 2010-03-08 21:44
I think this bug still exists in Python 2.6.4, and I haven't tested 2.6.5, but since 2.6.4 was released 6 months after this bug was closed I assume it's still an issue. Running ndbecker's test program as is produces the following output on Solaris 10:

Process PoolWorker-1:
Process PoolWorker-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
Process PoolWorker-3:
Traceback (most recent call last):
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    self.run()
    task = get()
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    self._target(*self._args, **self._kwargs)
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument
    task = get()
  File "/opt/app/g++lib6/python-2.6/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
TypeError: type 'partial' takes at least one argument
msg130511 - (view) Author: (uptimebox) Date: 2011-03-10 16:21
The bug still exists in 2.6.6

Process PoolWorker-2:
Traceback (most recent call last):
Process PoolWorker-1:
Traceback (most recent call last):
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
  File "/usr/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
    self.run()
    self.run()
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
  File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run
    self._target(*self._args, **self._kwargs)
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 57, in worker
    task = get()
    task = get()
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
  File "/usr/lib/python2.6/multiprocessing/queues.py", line 352, in get
    return recv()
    return recv()
TypeError: type 'partial' takes at least one argument
TypeError: type 'partial' takes at least one argument
msg130513 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-03-10 16:49
The bug was fixed in 2.7, and not backported to 2.6.
2.6 is now in security-fix-only mode, if possible you should upgrade to 2.7.
msg130548 - (view) Author: (uptimebox) Date: 2011-03-11 07:29
Unfortunately, upgrading to 2.7 is not an option for me, since my application works on Debian stable which only includes 2.6. For now I'm settled with the following workaround: http://paste.pocoo.org/show/351774/
History
Date User Action Args
2011-04-03 21:39:03jwilksetnosy: + jwilk
2011-03-11 07:29:36uptimeboxsetnosy: amaury.forgeotdarc, pitrou, jackdied, christian.heimes, jnoller, ndbecker, joseph.h.garvin, uptimebox
messages: + msg130548
2011-03-10 16:49:43amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg130513
2011-03-10 16:21:12uptimeboxsetnosy: + uptimebox
messages: + msg130511
2010-03-08 21:44:37joseph.h.garvinsetnosy: + joseph.h.garvin

messages: + msg100669
versions: + Python 2.6, - Python 2.5
2009-03-31 23:48:06jackdiedsetstatus: open -> closed
resolution: fixed
messages: + msg84949
2009-03-31 19:19:12jnollersetassignee: jnoller -> jackdied

messages: + msg84872
nosy: + jackdied
2009-03-31 19:13:26jnollersetmessages: + msg84871
2009-03-29 14:37:58jnollersetpriority: low
2009-02-12 18:03:27jnollersetmessages: + msg81789
2009-02-12 18:01:51pitrousetnosy: + pitrou
messages: + msg81788
2009-02-12 14:28:26jnollersetmessages: + msg81767
2009-02-12 14:27:56jnollersetassignee: jnoller
nosy: + christian.heimes
2009-02-12 14:26:28ndbeckercreate