This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Can not use append/extend to lists in a multiprocessing manager dict
Type: behavior Stage:
Components: Documentation Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: jnoller Nosy List: Jimbofbx, alex, asksol, georg.brandl, jnoller, pitrou
Priority: normal Keywords:

Created on 2010-09-08 19:23 by Jimbofbx, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (7)
msg115891 - (view) Author: James Hutchison (Jimbofbx) Date: 2010-09-08 19:23
tested python 3.1.2

Man = multiprocessing.Manager();
d = man.dict();
d['l'] = list();
d['l'].append("hey");
print(d['l']);
>>> []

using debugger reveals a KeyError. Extend also does not work. Only thing that works is += which means you can't insert actual tuples or lists into the list. This was all done on a single process
msg115919 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-09-08 23:45
Similarly:

>>> x = man.list()
>>> x.append({})
>>> x[0]
{}
>>> x[0]['a'] = 5
>>> x[0]
{}

Lots of fun!
msg117150 - (view) Author: Ask Solem (asksol) (Python committer) Date: 2010-09-22 18:41
Maybe surprising but not so weird if you think about what happens
behind the scenes.

When you do

    >>> x = man.list()
    >>> x.append({})

You send an empty dict to the manager to be appended to x

when do:

   >>> x[0]
   {}

you receive a local copy of the empty dict from the manager process.


So this:

    >>> x[0]["a"] = 5

will only modify the local copy.

What you would have to do is:

    >>> x.append({})
    >>> t = x[0]
    >>> t["a"] = 5
    >>> x[0] = t

This will not be atomic of course, so this may be something
to take into account.

What maybe could be supported is something like:
    >>> x[0] = manager.dict()
    >>>x[0]["foo"] = "bar"

but otherwise I wouldn't consider this a bug.
msg117166 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2010-09-23 03:54
It should be documented though.  Similar scenario in the Django docs: http://docs.djangoproject.com/en/1.2/topics/http/sessions/#when-sessions-are-saved
msg117172 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-09-23 06:47
Changing to doc issue.
msg117202 - (view) Author: James Hutchison (Jimbofbx) Date: 2010-09-23 15:55
Is there a way to get this so it behaves more intuitively? You'd think adding a managed list to a managed dictionary (or another managed list) or making a deep copy would work but it still doesn't. When you get an item from a managed data structure, it seems to be returning a data-only copy of the object instead of a handle to the manager of the object. The fact that += (extend) works but .extend() doesn't work also seems to raise a flag for me (although I do understand why this is). I don't think it should behave this way.

i.e.:
currently:
d['l'] -> return copy.deepcopy(d['l'])

should be:
d['l'] -> return managerObject(d['l'])
where managerObject is a managed object that runs on the same process as the manager it came from

Problem: Currently there is no easy way to do random access without copying out and copying back in. I'd think that would be a real efficiency problem.
msg118795 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-10-15 16:20
For now, documented the current behavior in r85534.  If a different solution is desired, a new issue can be opened.
History
Date User Action Args
2022-04-11 14:57:06adminsetgithub: 54010
2010-10-15 16:20:28georg.brandlsetstatus: open -> closed
resolution: wont fix
messages: + msg118795
2010-09-23 15:55:45Jimbofbxsetmessages: + msg117202
2010-09-23 06:47:57georg.brandlsetnosy: + georg.brandl
messages: + msg117172
components: + Documentation, - Library (Lib)
2010-09-23 03:54:49alexsetnosy: + alex
messages: + msg117166
2010-09-22 18:41:27asksolsetnosy: + asksol
messages: + msg117150
2010-09-08 23:45:38pitrousetnosy: + pitrou
messages: + msg115919
2010-09-08 23:43:54pitrousetassignee: jnoller

nosy: + jnoller
versions: + Python 3.1, Python 2.7, Python 3.2
2010-09-08 19:23:20Jimbofbxcreate