classification
Title: Error pickling objects with mutating __getstate__
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Leo.Na, alexandre.vassalotti, belopolsky, docs@python, neologix, pitrou, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2010-12-14 14:53 by belopolsky, last changed 2017-02-22 05:29 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
dict-graph.py belopolsky, 2010-12-14 14:53
issue10701.py alexandre.vassalotti, 2013-04-14 05:03
Messages (6)
msg123948 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-12-14 14:53
The work-around that I proposed for issue10700 does not work with Python 2.x:

$ python2.7 dict-graph.py
Vertex 0 -> 2, 1
Vertex 1 -> 
Vertex 2 -> 
Traceback (most recent call last):
  File "dict-graph.py", line 74, in <module>
    p = pickle.dumps(g)
    ...
File ".../Lib/pickle.py", line 661, in _batch_setitems
    for k, v in items:
RuntimeError: dictionary changed size during iteration
msg180046 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-01-15 20:06
It's interesting. The example behaves unstable on 3.3+ with C implementation of picle, sometimes works, sometimes fails. With Python implementation and on 3.2 it works always. On 2.7 it fails always.

A difference between C and Python implementations of pickle is a bug and should be fixed.
msg180048 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-15 20:16
I find the posted code mystifying. As the name suggests, __getstate__ should probably not mutate anything.
It would be nice if you could post a simpler example to reproduce issue. Even better if it doesn't have a mutating __getstate__, I would say.
msg180088 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-01-16 13:07
> It's interesting. The example behaves unstable on 3.3+ with C implementation of picle, sometimes works, sometimes fails. With Python implementation and on 3.2 it works always. On 2.7 it fails always.

That's hash randomization.
msg186882 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-04-14 05:03
The mutating __getstate__ is very likely the problem here. I've attached a small test case which shows the described behavior.

We could fix this by always making a copy of any mutable container we  want to iterate over to save its items. Performance-wise this solution is not very attractive. We are better off documenting this behavior as a limitation of __getstate__.
msg288262 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-02-21 04:07
I agree with Antoine that, "As the name suggests, __getstate__ should probably not mutate anything."

Unless a problematic non-mutating example can be found, I suggest this be closed.  For the most part, our rule has been that pure python code doesn't have to (and possibly cannot) defend itself against mid-stream mutation, while C code only has to defend itself to the point of avoiding a segfault.  

IMO, "RuntimeError: dictionary changed size during iteration" is an informative error message in this case.
History
Date User Action Args
2017-02-22 05:29:18serhiy.storchakasetstatus: open -> closed
resolution: not a bug
stage: needs patch -> resolved
2017-02-21 04:07:46rhettingersetstatus: pending -> open
nosy: + rhettinger
messages: + msg288262

2017-02-19 14:24:29serhiy.storchakasetstatus: open -> pending

nosy: + docs@python
versions: + Python 3.5, Python 3.6, Python 3.7, - Python 3.3, Python 3.4
assignee: docs@python
components: + Documentation
type: behavior -> enhancement
2013-12-06 05:13:09alexandre.vassalottisettitle: Error pickling a dict -> Error pickling objects with mutating __getstate__
2013-04-14 05:03:39alexandre.vassalottisetfiles: + issue10701.py

messages: + msg186882
2013-01-16 13:07:33neologixsetnosy: + neologix
messages: + msg180088
2013-01-15 20:16:25pitrousetmessages: + msg180048
2013-01-15 20:06:37serhiy.storchakasetnosy: + alexandre.vassalotti, serhiy.storchaka, pitrou

messages: + msg180046
versions: + Python 3.3, Python 3.4
2010-12-14 14:53:54belopolskycreate