Title: pickle - cannot unpickle circular deps with custom __hash__
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.0, Python 3.1, Python 2.6
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, collinwinter, edonmyder, swiftcoder
Priority: normal Keywords:

Created on 2007-07-26 12:36 by edonmyder, last changed 2009-06-29 06:33 by alexandre.vassalotti. This issue is now closed.

File name Uploaded Description Edit edonmyder, 2007-07-26 12:36 minimal example
Messages (4)
msg32560 - (view) Author: Martin Süßkraut (edonmyder) Date: 2007-07-26 12:36
unpickle a object with a custom __hash__ fails.

Example (full code is attached):

class Node(object):
    def __init__(self, i):
	self.i = i

    def __cmp__(self, other):
	return cmp(self.i, other.i)

    def __hash__(self):
	return hash(self.i)

n = Node(12)
n.next_nodes = set((n,))

unpickling n give the following error message:
Traceback (most recent call last):
  File "", line 23, in <module>
    n1 = pickle.load(f1)
  File "/usr/lib/python2.5/", line 1370, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.5/", line 858, in load
  File "/usr/lib/python2.5/", line 1133, in load_reduce
    value = func(*args)
  File "", line 15, in __hash__
    return hash(self.i)
AttributeError: 'Node' object has no attribute 'i'
msg58060 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2007-12-01 17:58
Please assign this bug to me.

This certainly doesn't look easy to fix. I will look into it, but I can
promise that I can fix it.
msg88079 - (view) Author: Tristam MacDonald (swiftcoder) Date: 2009-05-19 15:39
Is there any sign of a patch or workaround for this issue?
msg88324 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-05-25 21:14
Checked this out more throughly and I came to the conclusion this cannot
be fixed without a considerable amount of work.

The problem is pickle adds an Node instance stub in the next_nodes set
before its attributes are ready. Since the stub doesn't have any
attribute at the time its added to the set, the __hash__ method fails
with an AttributeError exception.

To fix this, pickle would need to detect cyclic objects with a custom
__hash__ method; and when it would see one, it would need to emit POP
opcodes to revert the parts of the object already pickled. And then,
pickle would have to re-pickle the cyclic object using a special
procedure that would delay the use of __hash__ until all the attributes
of the object are ready to be used.

I do not believe the bug arises frequently enough to justify adding more
tricky code to pickle. So, I will not fix this myself (feel free to
write a patch, however).

Finally, you can workaround the bug using the __getstate__/__setstate__
mechanism as follow:

class Node(object):
  def __init__(self, i):
    self.i = i
    self.next_nodes = set()
  def __cmp__(self, other):
	return cmp(self.i, other.i)
  def __hash__(self):
    return hash(self.i)
  def __getstate__(self):
    next_nodes = self.next_nodes.copy()
    return {'i': self.i,
            'self_in_next_nodes': self in self.next_nodes,
            'next_nodes': next_nodes}
  def __setstate__(self, state):
    if state.pop('self_in_next_nodes'):

n = Node(12)
n.next_nodes = set([n])
Date User Action Args
2009-06-29 06:33:16alexandre.vassalottisetstatus: pending -> closed
2009-05-25 21:14:06alexandre.vassalottisetstatus: open -> pending
assignee: alexandre.vassalotti ->
resolution: wont fix
messages: + msg88324
2009-05-19 15:39:35swiftcodersetmessages: + msg88079
2009-05-19 15:37:06swiftcodersetnosy: + swiftcoder
2009-04-06 10:51:24ajaksu2setnosy: + collinwinter
versions: + Python 3.1, - Python 2.5

stage: needs patch
2007-12-01 18:02:11alexandre.vassalottisetassignee: alexandre.vassalotti
2007-12-01 17:58:43alexandre.vassalottisetnosy: + alexandre.vassalotti
type: behavior
messages: + msg58060
versions: + Python 2.6, Python 3.0
2007-07-26 12:36:54edonmydercreate