This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: pickle + weakref.proxy(self)
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: alexandre.vassalotti, phd, pitrou, serhiy.storchaka
Priority: normal Keywords:

Created on 2013-05-26 18:59 by phd, last changed 2022-04-11 14:57 by admin.

Messages (3)
msg190105 - (view) Author: Oleg Broytman (phd) * Date: 2013-05-26 18:59
Hello! I've found a problematic behaviour of pickle when it pickles
(direct or indirect) weak proxies to self. I suspect pickle cannot
detect reference loops with weak proxies. I managed to narrow the case
to the following example:

import pickle, weakref

class TestPickle(object):
    def __init__(self):
        self.recursive = weakref.proxy(self)

    def __getstate__(self):
        print "__getstate__", id(self)
        return self.__dict__.copy()

    def __setstate__(self, d):
        print "__setstate__", id(self)
        self.__dict__.update(d)

print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"

   It prints:

- 1 -
- 2 -
__getstate__ 3075348620
__getstate__ 3075348620
- 3 -
__setstate__ 3075348844
__setstate__ 3075349004
- 4 -
Result: 3075349004
- 5 -

   That is, __getstate__ is called twice for the same object. And what
is worse, __setstate__ is called twice for different objects. The
resulting unpickled object is the last one, but in the library that I
have been debugging creation of two different objects during unpickling
is a bug.

   I can fix it by avoiding pickling the proxy and recreating the proxy
on unpickling:

import pickle, weakref

class TestPickle(object):
    def __init__(self):
        self.recursive = weakref.proxy(self)

    def __getstate__(self):
        print "__getstate__", id(self)
        d = self.__dict__.copy()
        del d['recursive']
        return d

    def __setstate__(self, d):
        print "__setstate__", id(self)
        self.__dict__.update(d)
        self.recursive = weakref.proxy(self)

print "- 1 -"
test = TestPickle()
print "- 2 -"
data = pickle.dumps(test, pickle.HIGHEST_PROTOCOL)
print "- 3 -"
test2 = pickle.loads(data)
print "- 4 -"
print "Result:", id(test2)
print "- 5 -"

- 1 -
- 2 -
__getstate__ 3075070092
- 3 -
__setstate__ 3075070188
- 4 -
Result: 3075070188
- 5 -

   But I wonder if it's a bug that should be fixed? If it's an expected
behaviour it perhaps should be documented as a warning in docs for
pickle or weakref or both.
msg199902 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-14 15:01
I don't think weakref objects are meant to be picklable at all. At least there's no support code for that, so pickling weak proxies only works by chance.
msg255541 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-28 14:11
The weakref.proxy object delegates all attribute access to referred object, including special attributes used in pickling, like __class__ or __getstate__.

test.recursive is not test, but it looks as TestPickle instance for the pickler. It is pickled and unpickled as TestPickle instance.

>>> test.recursive.__class__
<class '__main__.TestPickle'>
>>> test.recursive.__getstate__
<bound method TestPickle.__getstate__ of <__main__.TestPickle object at 0xb7130bac>>
>>> test.recursive.__reduce_ex__(2)
__getstate__ 3071478700
(<function __newobj__ at 0xb74b76f4>, (<class '__main__.TestPickle'>,), {'recursive': <weakproxy at 0xb7127a04 to TestPickle at 0xb7130bac>}, None, None)
>>> pickle.loads(pickle.dumps(test.recursive, 2))
__getstate__ 3071478700
__setstate__ 3071525356
<__main__.TestPickle object at 0xb713c1ec>

Since test.recursive.recursive is test.recursive, this recursion is detected by pickler.

This issue is similar to issue6395, but since weakref.proxy is not subclassable, we can just implement correct __reduce__.
History
Date User Action Args
2022-04-11 14:57:46adminsetgithub: 62268
2015-11-28 14:11:50serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg255541

assignee: serhiy.storchaka
stage: needs patch
2013-10-14 15:01:51pitrousetversions: + Python 3.4, - Python 2.6, Python 2.7
nosy: + alexandre.vassalotti

messages: + msg199902

type: behavior -> enhancement
2013-10-14 14:26:20georg.brandlsetnosy: + pitrou
2013-05-26 18:59:09phdcreate