classification
Title: pickle bug - recursively memoizing class?
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.2
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Cannot pickle self-referencing sets
View: 9269
Assigned To: belopolsky Nosy List: alexandre.vassalotti, belopolsky, grubert, schmir, zzzeek
Priority: normal Keywords:

Created on 2004-07-27 20:55 by skip.montanaro, last changed 2010-08-16 20:02 by belopolsky. This issue is now closed.

Files
File name Uploaded Description Edit
picklecycle.py skip.montanaro, 2004-07-27 20:55
memoize.py skip.montanaro, 2004-07-27 20:56
cycle.out skip.montanaro, 2004-07-27 20:57
picklecycle3.py belopolsky, 2010-07-15 03:54
Messages (8)
msg21827 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2004-07-27 20:55
The attached script (picklecycle.py) gives me an
assertion error
in Pickler.memoize().  Maybe I'm just doing something dumb.
I have very little experience with pickle and even less
experience
pickling new-style class instances.  I'm just trying to
adapt an
existing messy object graph to allow it to be pickled.
 I realize the
two instances refer to each other, but I thought pickle was
supposed to gracefully handle cyclic object graphs.

I can worm around the problem a couple ways.  First,
getting
rid of new-style classes does the trick.  Also
modifying B.__reduce__
to not return self.__dict__ seems to do the trick.

Output (cycle.out) and a modified version of
Pickler.memoize
(memoize.py) will be attached momentarily.
msg21828 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2004-07-27 20:56
Logged In: YES 
user_id=44345

Attaching chatty version of Pickler.memoize()
msg21829 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2004-07-27 20:57
Logged In: YES 
user_id=44345

Attaching the output I see when running picklecycle.py w/ the
modified memoize() method (cycle.out).
msg21830 - (view) Author: engelbert gruber (grubert) * Date: 2005-05-19 11:00
Logged In: YES 
user_id=147070

it does memoize recursively, 
but "asserts" not to get stuck in cycles.

that's how i understand the code if ::

        assert id(obj) not in self.memo

is replaced by ::

        if id(obj) in self.memo:
            return

it works (better). and test_pickle.py still passes.
msg77200 - (view) Author: mike bayer (zzzeek) * Date: 2008-12-07 01:03
This bug can be reproduced very easily:

    import pickle

    class MyClass(object):
        pass
    
    m = MyClass()
    m2 = MyClass()

    s = set([m])
    m.foo = set([m2])
    m2.foo = s

    print pickle.dumps(s)

This bug is critical as the pure-python pickle module is required when
trying to remove warnings in -3 mode with Python 2.6, which is something
that will be very common in the near future.   It's easily reproducible
with simplistic object cycles like that of the above, in 2.5, 2.6 and
3.0 at least.
msg88576 - (view) Author: mike bayer (zzzeek) * Date: 2009-05-30 19:07
im noticing my test case seems to work fine in py 3.0.1.   haven't
tested 2.6.2.
msg110343 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-15 03:54
This looks like a duplicate of issue 1581183.  The issue is still present in py3k, but only shows up in picklecycle.py when using pickle.py version. I am attaching picklecycle3.py which is py3k version of picklecycle.py using python pickler.
msg110400 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-07-15 23:10
There are two issues here.  The original issue is a duplicate of #1062277.  The other issue is specific to cycles containing a set. [msg77200]

I am opening a separate issue for that. See #9269.
History
Date User Action Args
2010-08-16 20:02:38belopolskysetstatus: pending -> closed
2010-07-15 23:10:13belopolskysetstatus: open -> pending
resolution: duplicate
superseder: Cannot pickle self-referencing sets
messages: + msg110400
2010-07-15 03:54:06belopolskysetfiles: + picklecycle3.py

type: behavior
assignee: belopolsky
versions: + Python 3.2
nosy: + alexandre.vassalotti, belopolsky

messages: + msg110343
stage: needs patch
2010-05-20 20:30:12skip.montanarosetnosy: - skip.montanaro
2009-05-30 19:07:04zzzeeksetmessages: + msg88576
2008-12-07 01:03:34zzzeeksetnosy: + zzzeek
messages: + msg77200
2008-03-14 17:42:36schmirsetnosy: + schmir
2004-07-27 20:55:41skip.montanarocreate