Hello! I've found a problematic behaviour of pickle when it pickles
(direct or indirect) weak proxies to self. I suspect pickle cannot
detect reference loops with weak proxies. I managed to narrow the case
to the following example:
import pickle, weakref
class TestPickle(object):
def __init__(self):
self.recursive = weakref.proxy(self)
def __getstate__(self):
print "__getstate__", id(self)
return self.__dict__.copy()
def __setstate__(self, d):
print "__setstate__", id(self)
self.__dict__.update(d)
print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"
It prints:
- 1 -
- 2 -
__getstate__ 3075348620
__getstate__ 3075348620
- 3 -
__setstate__ 3075348844
__setstate__ 3075349004
- 4 -
Result: 3075349004
- 5 -
That is, __getstate__ is called twice for the same object. And what
is worse, __setstate__ is called twice for different objects. The
resulting unpickled object is the last one, but in the library that I
have been debugging creation of two different objects during unpickling
is a bug.
I can fix it by avoiding pickling the proxy and recreating the proxy
on unpickling:
import pickle, weakref
class TestPickle(object):
def __init__(self):
self.recursive = weakref.proxy(self)
def __getstate__(self):
print "__getstate__", id(self)
d = self.__dict__.copy()
del d['recursive']
return d
def __setstate__(self, d):
print "__setstate__", id(self)
self.__dict__.update(d)
self.recursive = weakref.proxy(self)
print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"
- 1 -
- 2 -
__getstate__ 3075070092
- 3 -
__setstate__ 3075070188
- 4 -
Result: 3075070188
- 5 -
But I wonder if it's a bug that should be fixed? If it's an expected
behaviour it perhaps should be documented as a warning in docs for
pickle or weakref or both.
|