This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: problem with pickling newstyle class instances
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: ajaksu2, alexandre.vassalotti, georg.brandl, python-dev, serhiy.storchaka, tovrstra
Priority: normal Keywords: patch

Created on 2004-02-08 15:55 by tovrstra, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
pickletest.py tovrstra, 2004-02-08 15:55 pickletest.py
pickle_recursive-2.7.patch serhiy.storchaka, 2015-10-25 19:46 review
pickle_recursive-2.7_2.patch serhiy.storchaka, 2015-11-05 08:08 review
pickle_recursive-2.7_3.patch serhiy.storchaka, 2015-11-06 13:32 review
Messages (13)
msg19940 - (view) Author: Toon Verstraelen (tovrstra) Date: 2004-02-08 15:55
My python version:

Python 2.3.3 (#1, Jan 25 2004, 21:45:01)
[GCC 3.2.3 20030422 (Gentoo Linux 1.4 3.2.3-r3,
propolice)] on linux2

Please try the example. That will explain a lot. The
problem is that not all new style class instances are
picklable. In my example a class is derived from a list
and a base class. The list-descendant contains an
instance of the base class, which has a reference to
the list containing it. with cPickle things work fine,
but not for the normal pickle routines

class subitem:
	def __init__(self, parent):
		self.parent = parent
		if parent != None:
			parent.append(self)

class group(subitem, list):
	def __init__(self, parent):
		subitem.__init__(self, parent)
		

g = group(None)
s = subitem(g)

import cPickle
print cPickle.dumps(g)

import pickle
print pickle.dumps(g)
msg19941 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2005-10-01 13:40
Logged In: YES 
user_id=1188172

Verified in 2.5cvs.
msg58057 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2007-12-01 17:35
Please assign this bug to me.

Note that neither cPickle or pickle is able to load the stream generated
by cPickle correctly:

   >>> g = group(None)
   >>> subitem(g)
   >>> g[0].parent is g
   True
   >>> gp = cPickle.loads(cPickle.dumps(g))
   >>> gp[0].parent is gp
   False

I don't think that will be easy to fix, but I will try to see what I can do.
msg82010 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-02-14 11:31
Confirmed on trunk.
msg114318 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-19 05:38
This is still a problem with 2.6 and 2.7.  I don't know how to test this with py3k as cPickle doesn't exist, can someone advise, thanks.
msg188288 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-05-03 01:33
I fixed this while working on PEP 3154 [http://hg.python.org/features/pep-3154-alexandre/rev/eed9142d664f]. The relevant piece is

@@ -420,7 +424,13 @@ class _Pickler:
             write(REDUCE)
 
         if obj is not None:
-            self.memoize(obj)
+            # If the object is already in the memo, this means it is
+            # recursive. In this case, throw away everything we put on the
+            # stack, and fetch the object back from the memo.
+            if id(obj) in self.memo:
+                write(POP + self.get(self.memo[id(obj)][0]))
+            else:
+                self.memoize(obj)
 
         # More new special cases (that work with older protocols as
         # well): when __reduce__ returns a tuple with 4 or 5 items,

It would be pretty easy to backport this to 2.7 and 3.3. It is also good to mention that that only protocol 0 and 1 are affected.
msg253435 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-10-25 19:46
Here is a patch that backports recursive objects handling to 2.7.
msg254003 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-03 16:53
Could you please make a review Alexandre?
msg254091 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-05 08:08
Updated patch addresses Alexandre's comments.
msg254191 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-06 13:32
Here is revised patch that uses different approach to tests. cPickleFastPicklerTests overridden old recursive tests to check that they raises an exception. The patch extends this to new recursive tests.
msg254264 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-07 09:17
New changeset 9ad1fd251ddf by Serhiy Storchaka in branch '2.7':
Issue #892902: Fixed pickling recursive objects.
https://hg.python.org/cpython/rev/9ad1fd251ddf

New changeset 2071d16ed5e6 by Serhiy Storchaka in branch '3.4':
Issue #892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/2071d16ed5e6

New changeset f33ce913220b by Serhiy Storchaka in branch '3.5':
Issue #892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/f33ce913220b

New changeset 2c81a883d8ca by Serhiy Storchaka in branch 'default':
Issue #892902: Added new tests for pickling recursive collections.
https://hg.python.org/cpython/rev/2c81a883d8ca
msg254265 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-11-07 09:22
Thank you for your review Alexandre.

In 3.x old tests test_recursive_set and test_recursive_frozenset now are implemented in test_recursive_set_and_inst and test_recursive_frozenset_and_inst. Instead new test_recursive_set now tests protocol 4 ability of pickling recursive sets itself.
msg254291 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-11-07 18:05
New changeset 77184a429dae by Serhiy Storchaka in branch '2.7':
Issue #892902: Disable newly added tests in test_xpickle.
https://hg.python.org/cpython/rev/77184a429dae
History
Date User Action Args
2022-04-11 14:56:02adminsetgithub: 39916
2015-11-07 18:05:43python-devsetmessages: + msg254291
2015-11-07 09:22:53serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg254265

stage: patch review -> resolved
2015-11-07 09:17:41python-devsetnosy: + python-dev
messages: + msg254264
2015-11-06 13:32:48serhiy.storchakasetfiles: + pickle_recursive-2.7_3.patch

messages: + msg254191
2015-11-05 08:08:54serhiy.storchakasetfiles: + pickle_recursive-2.7_2.patch

messages: + msg254091
2015-11-03 16:53:57serhiy.storchakasetmessages: + msg254003
2015-10-25 19:46:21serhiy.storchakasetfiles: + pickle_recursive-2.7.patch

nosy: + serhiy.storchaka
messages: + msg253435

keywords: + patch
stage: needs patch -> patch review
2015-10-25 10:30:23serhiy.storchakasetstage: test needed -> needs patch
versions: - Python 3.1, Python 3.2
2014-02-03 18:31:31BreamoreBoysetnosy: - BreamoreBoy
2013-05-03 01:33:11alexandre.vassalottisetmessages: + msg188288
2010-08-19 05:39:46BreamoreBoysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2010-08-19 05:38:48BreamoreBoysetnosy: + BreamoreBoy
messages: + msg114318
2009-02-14 11:31:46ajaksu2setnosy: + ajaksu2
stage: test needed
messages: + msg82010
versions: + Python 2.6, - Python 2.3
2007-12-01 18:00:25alexandre.vassalottisetassignee: alexandre.vassalotti
2007-12-01 17:35:21alexandre.vassalottisetnosy: + alexandre.vassalotti
type: behavior
messages: + msg58057
components: + Library (Lib), - Interpreter Core
2004-02-08 15:55:57tovrstracreate