classification
Title: Cannot modify dictionaries inside dictionaries using Managers from multiprocessing
Type: behavior Stage: needs patch
Components: Versions: Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: jnoller Nosy List: Richard.Fothergill, carlosdf, dariosg, jnoller, kghose, terrence
Priority: normal Keywords: patch

Created on 2009-08-23 17:46 by carlosdf, last changed 2014-03-14 11:20 by Richard.Fothergill.

Files
File name Uploaded Description Edit
mp_proxy_hack.diff terrence, 2009-10-14 00:33 a crude hack that appears to work review
test_dict_dict_arrays.py dariosg, 2011-04-13 08:59
nesting.py Richard.Fothergill, 2014-03-14 11:20 Code example from comment
Messages (10)
msg91889 - (view) Author: Carlos (carlosdf) Date: 2009-08-23 17:46
It's not possible to modify a dict inside a dict using a manager from 
multiprocessing.

Ex:

from multiprocessing import Process,Manager

def f(d):
    d['1'] = '1'
    d['2']['1'] = 'Try To Write'

if __name__ == '__main__':
    manager = Manager()

    d = manager.dict()

    d['2'] = manager.dict()

    print d

    p = Process(target=f, args=(d,))
    p.start()
    p.join()

    print d

    d['2'] = 5
    print d

The output Under Windows 7 (32 Bits) / Python 2.6.2 (32 Bits) is:

{'2': {}}
{'1': '1', '2': {}}
{'1': '1', '2': 5}

The output is the same if you change "d['2'] = manager.dict()" to 
"d['2'] = dict()"
msg93948 - (view) Author: Terrence Cole (terrence) Date: 2009-10-13 23:28
I get the same results on:
Python 2.6.2 (r262:71600, Sep 14 2009, 18:47:57)
[GCC 4.3.2] on linux2

I think this is the same issue I was seeing yesterday.  You can exercise
the issue and cause an exception with just 6 lines:

##### CODE #####
from multiprocessing import Manager
manager = Manager()
ns_proxy = manager.Namespace()
evt_proxy = manager.Event()
ns_proxy.my_event_proxy = evt_proxy
print ns_proxy.my_event_proxy

##### TRACEBACK #####
Traceback (most recent call last):
  File "test_nsproxy.py", line 39, in <module>
    print ns_proxy.my_event_proxy
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 989, in
__getattr__
    return callmethod('__getattribute__', (key,))
  File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in
_callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------
Unserializable message: ('#RETURN', <threading._Event object at 0x1494790>)
---------------------------------------------------------------------

Storing a proxy into a proxied object and then accessing the proxy 
returns a copy of the object itself and not the stored proxy.  Thus,
updates to the nested dict are local and do not update the real object,
and proxies to unpicklable objects raise an exception when accessed.
msg93951 - (view) Author: Terrence Cole (terrence) Date: 2009-10-14 00:33
When a manager receives a message, it unpickles the arguments; this
calls BaseProxy.__reduce__, which calls RebuildProxy.  If we are in the
manager, this returns the actual object, otherwise it returns a new
proxy.  If we naively disable the ability for proxied objects to be
unredirected in the manager, as in the attached svn diff, this solves
the problem that Carlos and I are seeing.  Surprisingly, after applying
this change, the full multiprocessing regression test still runs fine. 
I'm sure this change should have some greater impact, but I'm not sure
what.  I would appreciate if someone more knowledgeable could comment.
msg93957 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-10-14 01:40
Nothing jumps out to me off the top of my head - I can take a closer look 
at this after my pycon planning duties finish up in a few weeks. I agree 
this is unintended behavior. I'll need to audit the tests to make sure 
that A> This is being tested, and B> Those tests are not disabled.

When we included multiprocessing, some tests were deemed too unstable at 
the time, and we disabled. This was unfortunate, and I haven't been able 
to circle back and spend the time needed to refactor the test suite.
msg93961 - (view) Author: Terrence Cole (terrence) Date: 2009-10-14 02:16
The tests for the SyncManager are being automagically generated at
import time -- I was not quite able to follow that well enough to know
exactly what is getting tested, or if they are even enabled.  It did not
appear to contain any recursion, however.
msg93962 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2009-10-14 02:17
Yeah, the auto-generation is too clever and needs to be pulled out 
entirely.
msg98529 - (view) Author: Kaushik Ghose (kghose) Date: 2010-01-29 19:17
Even with the patch, I can not resolve this problem. I can reproduce the problem with the patched version with the following code. My system is:

Python 2.6.4 (r264:75821M, Oct 27 2009, 19:48:32)
IPython 0.10
Platform is Mac OS X (10.5.8) Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009

import multiprocessing as mp

def f(d):
  d['f'] = {}
  d['f']['msg'] = 'I am here'

manager = mp.Manager()
d = manager.dict()

p = mp.Process(target=f, args=(d,))

p.start()
p.join()

print d

d = {}
f(d)

print d

Output:

{'f': {}}
{'f': {'msg': 'I am here'}}
msg98548 - (view) Author: Terrence Cole (terrence) Date: 2010-01-30 04:06
Kaushik, in your example, d is a dict proxy, so assignment to d['f'] correctly ferries the assignment (a new normal dict) to the d['f'] in the original process.  The new dict, however, is not a dict proxy, it's just a dict, so assignment of d['f']['msg'] goes nowhere.  All hope is not lost, however, because the Manager can be forked to new processes.  The slightly modified example below shows how this works:

from multiprocessing import Process, Manager
def f(m, d):
    d['f'] = m.dict()
    d['f']['msg'] = 'I am here'

m = Manager()
d = m.dict()
p = Process(target=f, args=(m,d))
p.start()
p.join()
print d
{'f': <DictProxy object, typeid 'dict' at 0x7f1517902810>}
print d['f']
{'msg': 'I am here'}

With the attached patch, the above works as shown, without, it gives the same output as your original example.
msg133655 - (view) Author: Darío Suárez Gracia (dariosg) Date: 2011-04-13 08:59
Hello,
Trying to share a dictionary of dictionaries of lists with a manager I get the same problem with the patch applied in Python 2.7 (r27:82500, Nov 24 2010, 18:24:29) [GCC 4.1.2 20080704 (Red Hat 4.1.2-48)] on linux2.

The shared variable in results and what I'm trying to do is simultaneously parsing multiple files.

The quality of the code is not very good because I'm a newbie python programmer.

Best regards,
Darío
msg213534 - (view) Author: Richard Fothergill (Richard.Fothergill) Date: 2014-03-14 11:20
I'm getting these results on both:
Python 3.2.3 (default, Apr 10 2013, 06:11:55)
[GCC 4.6.3] on linux2
and
Python 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3] on linux2

The symptoms are exactly as Terrence described.

Nesting proxied containers is supposed to be a supported use case! From the documentation: http://docs.python.org/2/library/multiprocessing.html#proxy-objects

>>> a = manager.list()
>>> b = manager.list()
>>> a.append(b)         # referent of a now contains referent of b
>>> print a, b
[[]] []
>>> b.append('hello')
>>> print a, b
[['hello']] ['hello']

The documented code works as expected, but:
>>> a[0].append('world')  # Appends to b?
>>> print a, b
[['hello']] ['hello']

I've attached my reproduction as a script.
History
Date User Action Args
2014-03-14 11:20:44Richard.Fothergillsetfiles: + nesting.py
versions: + Python 3.2
nosy: + Richard.Fothergill

messages: + msg213534
2011-11-29 06:06:45ezio.melottisetstage: needs patch
versions: + Python 2.7, - Python 2.6
2011-04-13 08:59:34dariosgsetfiles: + test_dict_dict_arrays.py
nosy: + dariosg
messages: + msg133655

2010-01-30 04:06:20terrencesetmessages: + msg98548
2010-01-29 19:17:23kghosesetnosy: + kghose
messages: + msg98529
2009-10-14 02:17:27jnollersetmessages: + msg93962
2009-10-14 02:16:05terrencesetmessages: + msg93961
2009-10-14 01:40:04jnollersetmessages: + msg93957
2009-10-14 00:33:03terrencesetfiles: + mp_proxy_hack.diff
keywords: + patch
messages: + msg93951
2009-10-13 23:28:23terrencesetnosy: + terrence
messages: + msg93948
2009-08-23 18:59:19r.david.murraysetassignee: jnoller

nosy: + jnoller
2009-08-23 17:46:28carlosdfcreate