classification
Title: Python 2.6 can't read sets pickled with Python 3.0
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.0, Python 3.1, Python 2.7, Python 2.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Make pickle generated by Python 3.x compatible with 2.x and vice-versa.
View: 6137
Assigned To: Nosy List: ajaksu2, alexandre.vassalotti, collinwinter, hagen, pitrou, terry.reedy
Priority: critical Keywords: patch

Created on 2008-08-25 12:15 by hagen, last changed 2009-06-04 01:21 by alexandre.vassalotti. This issue is now closed.

Files
File name Uploaded Description Edit
pickle.diff ajaksu2, 2008-08-28 17:51 Adds a 3.0:2.6 modules mapping and uses that in find_class
Messages (10)
msg71916 - (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-25 12:15
After pickling a set of ints with Python 3.0 and pickle protocol 2:

[hagenf@chage ~]$ python3.0
Python 3.0b3 (r30b3:65927, Aug 21 2008, 11:48:29)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "wb")
>>> pickle.dump({1,2,3}, f, 2)
>>> f.close()

I get the following error when trying to read this with Python 2.6:

[hagenf@chage ~]$ python
Python 2.6b3 (r26b3:65922, Aug 21 2008, 11:42:25)
[GCC 4.1.0 20060304 (Red Hat 4.1.0-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> f = open("test", "rb")
>>> pickle.load(f)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1370, in load
    return Unpickler(file).load()
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 858, in load
    dispatch[key](self)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1090, in
load_global
    klass = self.find_class(module, name)
  File "/home/MC/hagenf/local/lib/python2.6/pickle.py", line 1124, in
find_class
    __import__(module)
ImportError: No module named builtins
msg72026 - (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-27 13:03
Well, this is obviously caused by renaming "__builtin__" to "builtins"
and the fact that set (as well as frozenset) doesn't have its own opcode
and therefore gets looked up in "builtins". The problem therefore
extends to all builtin objects without opcode special casing (e.g.
object, slice, property, ...) I'm afraid that means we have to pickle
"builtins" as "__builtin__" for backwards compatibility in protocols <= 2.

But aside from that, wouldn't it be more consistent to have opcodes for
set/frozenset in protocol 3?
msg72096 - (view) Author: Daniel Diniz (ajaksu2) Date: 2008-08-28 16:40
Hagen,
does this simple patch (against 2.6) solve it for you?

Index: Lib/pickle.py
===================================================================
--- Lib/pickle.py       (revision 66050)
+++ Lib/pickle.py       (working copy)
@@ -1121,6 +1121,8 @@

     def find_class(self, module, name):
         # Subclasses may override this
+        if module == "builtins":
+            module = "__builtin__"
         __import__(module)
         mod = sys.modules[module]
         klass = getattr(mod, name)

I think a dict mapping the moved modules would work better, perhaps
having it in PyPI would be enough?
msg72097 - (view) Author: Hagen Fürstenau (hagen) Date: 2008-08-28 16:52
Well, Python <= 2.5 still wouldn't be able to unpickle those built in
objects.
msg72100 - (view) Author: Daniel Diniz (ajaksu2) Date: 2008-08-28 17:51
FWIW, there's a mapping of 2.6:3.0 modules in lib2to3:

from lib2to3.fixes.fix_imports import MAPPING

The attached patch uses that for a quick and dirty way of loading 3.0
pickles in 2.6.
msg72164 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-08-29 21:54
Was it really intended that 3.0 pickles unpickle on 2.6?
What about other changes like moving something from one module to
another (reduce from built-in to functools), changing all classes to new
style (several examples), or adding methods to a built-in class (floats)?
msg72165 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2008-08-29 22:02
> Was it really intended that 3.0 pickles unpickle on 2.6?

He used protocol 2, so he explicitly asked for something inpickleable
with 2.6. If it's not the intended behaviour, then protocols < 3 should
be deprecated.
msg74328 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-10-04 23:14
I agree with Antoine, protocols <= 2 should remain compatible with
Python 2.x or be deprecated. Keeping compatibility will require a hack,
in addition to the proposed patch, in Pickler.save_global to map Python
3's module names to the ones of Python 2.
msg85299 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-04-03 08:10
Would it would be sufficient to add a function in the pickletools
capable of converting old Python 2 pickles to the new format in Python
3? Alternatively, we could add a compatibility Unpickler class that
would override the find_class method to do the name translation on-the-fly.
msg88862 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-04 01:21
Superseded by issue #6137.
History
Date User Action Args
2009-06-04 01:21:36alexandre.vassalottisetstatus: open -> closed
resolution: duplicate
superseder: Make pickle generated by Python 3.x compatible with 2.x and vice-versa.
messages: + msg88862
2009-06-03 22:39:37pitrouunlinkissue6137 dependencies
2009-06-03 22:32:48pitroulinkissue6137 dependencies
2009-04-04 00:31:27collinwintersetnosy: + collinwinter
2009-04-03 08:10:51alexandre.vassalottisetmessages: + msg85299
2009-03-31 13:10:10pitrousetversions: + Python 3.1, Python 2.7
2008-10-04 23:14:44alexandre.vassalottisetmessages: + msg74328
2008-08-29 22:02:11pitrousetnosy: + pitrou
messages: + msg72165
2008-08-29 21:54:36terry.reedysetnosy: + terry.reedy
messages: + msg72164
2008-08-28 17:51:42ajaksu2setfiles: + pickle.diff
keywords: + patch
messages: + msg72100
2008-08-28 16:52:05hagensetmessages: + msg72097
2008-08-28 16:41:00ajaksu2setnosy: + ajaksu2
messages: + msg72096
2008-08-27 13:03:24hagensetmessages: + msg72026
2008-08-25 15:58:24pitrousetpriority: critical
nosy: + alexandre.vassalotti
type: behavior
2008-08-25 12:15:41hagencreate