classification
Title: python 2.7.5 fails to unpickle namedtuple pickled by 2.7.3 or 2.7.4
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Arfrever, anselm.kruis, python-dev, rhettinger, schmir
Priority: normal Keywords: patch

Created on 2013-05-19 18:41 by anselm.kruis, last changed 2013-12-06 05:10 by alexandre.vassalotti. This issue is now closed.

Files
File name Uploaded Description Edit
nt_pickle_fix.diff rhettinger, 2013-05-20 02:28 Patch to fixup 2.7.6
fix_python_275_issue18015.pth anselm.kruis, 2013-05-22 11:36 Fixup collections.py for Python 2.7.5
Messages (4)
msg189614 - (view) Author: Anselm Kruis (anselm.kruis) * Date: 2013-05-19 18:41
Change 18303391b981 breaks unpickling named tuples pickled by 2.7.3 and 2.7.4.

See closed issue #15535 for the full story. Unfortunately Raymond was wrong, when he wrote that the addition of __dict__ was a 2.7.4 change. It was added by changeset 26d5f022eb1a in 2.7.3.

Now 2.7.5 can't unpickle any named tuples pickled by 2.7.3, which is probably one of the most widely used python versions.


Example:

Pickle a namd tuple using 2.7.3 and unpickle it using 2.7.5.

anselm@Emmy:~$ python2.7
Python 2.7.3 (default, Sep 16 2012, 21:46:37) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> import pickletools
>>> import pickle
>>> N=collections.namedtuple("N","a")
>>> n=N(1)
>>> p=pickle.dumps(n, 2)
>>> p2=pickletools.optimize(p)
>>> pickletools.dis(p2)
    0: \x80 PROTO      2
    2: c    GLOBAL     '__main__ N'
   14: K    BININT1    1
   16: \x85 TUPLE1
   17: \x81 NEWOBJ
   18: c    GLOBAL     'collections OrderedDict'
   43: ]    EMPTY_LIST
   44: ]    EMPTY_LIST
   45: (    MARK
   46: U        SHORT_BINSTRING 'a'
   49: K        BININT1    1
   51: e        APPENDS    (MARK at 45)
   52: a    APPEND
   53: \x85 TUPLE1
   54: R    REDUCE
   55: b    BUILD
   56: .    STOP
highest protocol among opcodes = 2
>>> print repr(p2)
'\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.'


anselm@Emmy:~/sc/eclipsews/fg2py$ fg2python
Python 2.7.5 (default, May 18 2013, 17:02:17) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import collections
>>> N=collections.namedtuple("N","a")
>>> pickle.loads('\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1419, in loads
    return Unpickler(file).load()
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 895, in load
    dispatch[key](self)
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1261, in load_build
    d = inst.__dict__
AttributeError: 'N' object has no attribute '__dict__'


As we can see from the trace back, the problem arises from the pickle op-code 'BUILD'. BUILD requires that the object to be build either has a method __setstate__ or has an attribute __dict__. Therefore I propose:

- Revert change 18303391b981 and add a __getstate__ method
  This is the Python 3 fix for the problem.

or

- Add a method __setstate__:

  def __setstate__(self, state): 
     """For compatibility with Python 2.7.3 and 2.7.4"""
     pass
msg189735 - (view) Author: Anselm Kruis (anselm.kruis) * Date: 2013-05-21 09:29
Just for the records: the patch works as expected.
msg189804 - (view) Author: Anselm Kruis (anselm.kruis) * Date: 2013-05-22 11:36
I created a small *.pth to monkey patch collections.py until 2.7.6 gets released. Maybe this is useful for someone else. Therefore I attach it here.

The pth file runs the following code during Python startup:

import collections
def _fix_issue_18015(collections):
    try:
        template = collections._class_template
    except AttributeError:
        # prior to 2.7.4 _class_template didn't exists
        return
    if not isinstance(template, basestring):
        return  # strange
    if "__dict__" in template or "__getstate__" in template:
        return  # already patched
    lines = template.splitlines()
    indent = -1
    for i,l in enumerate(lines):
        if indent < 0:
            indent = l.find('def _asdict')
            continue
        if l.startswith(' '*indent + 'def '):
            lines.insert(i, ' '*indent + 'def __getstate__(self): pass')
            lines.insert(i, ' '*indent + '__dict__ = _property(_asdict)')
            break
    collections._class_template = '''\n'''.join(lines)
_fix_issue_18015(collections)
msg190148 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2013-05-27 17:59
New changeset 687295c6c8f2 by Raymond Hettinger in branch '2.7':
Issue #18015: Fix unpickling of 2.7.3 and 2.7.4 namedtuples.
http://hg.python.org/cpython/rev/687295c6c8f2
History
Date User Action Args
2013-12-06 05:10:50alexandre.vassalottisetstatus: open -> closed
resolution: fixed
stage: resolved
2013-05-27 17:59:10python-devsetnosy: + python-dev
messages: + msg190148
2013-05-22 11:36:49anselm.kruissetfiles: + fix_python_275_issue18015.pth

messages: + msg189804
2013-05-21 09:32:35schmirsetnosy: + schmir
2013-05-21 09:29:00anselm.kruissetmessages: + msg189735
2013-05-20 02:28:35rhettingersetfiles: + nt_pickle_fix.diff
keywords: + patch
2013-05-19 23:04:56rhettingersetassignee: rhettinger
2013-05-19 22:33:51Arfreversetnosy: + Arfrever
2013-05-19 18:41:59anselm.kruiscreate