classification
Title: Document pickle behavior for subclasses of dicts/lists
Type: behavior Stage: needs patch
Components: Documentation Versions: Python 3.6, Python 3.5, Python 3.4, Python 2.7
process
Status: open Resolution:
Dependencies: 3635 Superseder:
Assigned To: docs@python Nosy List: alexandre.vassalotti, aronacher, eric.snow, georg.brandl, pitrou, rhettinger, serhiy.storchaka
Priority: normal Keywords: easy

Created on 2008-12-21 17:24 by georg.brandl, last changed 2015-11-26 22:11 by serhiy.storchaka.

Messages (4)
msg78154 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2008-12-21 17:24
When unpickling dict subclasses, the dict is filled via setitem before
__setstate__ is called.  This, and other behavior around subclasses of
classes that have special pickle behavior should be documented.
msg83098 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-03-03 21:04
The behavior is a PITA.  It means that dict subclasses the redefine
__setitem__ have unpleasant pickling challenges.  The __setitem__
insertions are called before the subclass can initialize.

The workaround involves a funky dance using __reduce__.  See
collections.OrderedDict::__reduce__() for an example.
msg255434 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-26 19:07
The copy module uses the same __reduce__ protocol, but reconstruct the object in different order, first set state, then add items. This discrepancy causes a difference between results of pickle/unpickle and copy operations. Example:

>>> class L(list):
...     def __getstate__(self):
...         return list(self)
...     def __setstate__(self, state):
...         self[:] = state
... 
>>> import copy, pickle
>>> pickle.loads(pickle.dumps(L([1, 2])))
[1, 2]
>>> copy.deepcopy(L([1, 2]))
[1, 2, 1, 2]

This was happened with xml.dom.minicompat.NodeList (issue10131).

Definitely one of pickle's or copy's behavior should be changed. But what?
msg255439 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-26 22:09
The order in copy was changed by issue1100562.
History
Date User Action Args
2015-11-26 22:11:25serhiy.storchakasetnosy: + pitrou, alexandre.vassalotti
2015-11-26 22:09:29serhiy.storchakasetmessages: + msg255439
2015-11-26 19:07:31serhiy.storchakasetnosy: + serhiy.storchaka

messages: + msg255434
versions: + Python 3.6
2014-04-02 04:07:26eric.snowsetnosy: + eric.snow

versions: + Python 3.5, - Python 3.2, Python 3.3
2012-07-25 22:53:49ezio.melottisetversions: + Python 3.2, Python 3.3, Python 3.4, - Python 2.6, Python 3.0, Python 3.1
2010-10-29 10:07:21adminsetassignee: georg.brandl -> docs@python
2009-05-16 22:56:32ajaksu2setdependencies: + pickle.dumps cannot save instance of dict-derived class that overrides __getattribute__
2009-04-22 14:39:15ajaksu2setkeywords: + easy
2009-03-03 21:04:39rhettingersetpriority: low -> normal
nosy: + rhettinger
messages: + msg83098
2008-12-21 17:24:04georg.brandlcreate