Change 18303391b981 breaks unpickling named tuples pickled by 2.7.3 and 2.7.4.

See closed issue #15535 for the full story. Unfortunately Raymond was wrong, when he wrote that the addition of __dict__ was a 2.7.4 change. It was added by changeset 26d5f022eb1a in 2.7.3.

Now 2.7.5 can't unpickle any named tuples pickled by 2.7.3, which is probably one of the most widely used python versions.


Example:

Pickle a namd tuple using 2.7.3 and unpickle it using 2.7.5.

anselm@Emmy:~$ python2.7
Python 2.7.3 (default, Sep 16 2012, 21:46:37) 
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import collections
>>> import pickletools
>>> import pickle
>>> N=collections.namedtuple("N","a")
>>> n=N(1)
>>> p=pickle.dumps(n, 2)
>>> p2=pickletools.optimize(p)
>>> pickletools.dis(p2)
    0: \x80 PROTO      2
    2: c    GLOBAL     '__main__ N'
   14: K    BININT1    1
   16: \x85 TUPLE1
   17: \x81 NEWOBJ
   18: c    GLOBAL     'collections OrderedDict'
   43: ]    EMPTY_LIST
   44: ]    EMPTY_LIST
   45: (    MARK
   46: U        SHORT_BINSTRING 'a'
   49: K        BININT1    1
   51: e        APPENDS    (MARK at 45)
   52: a    APPEND
   53: \x85 TUPLE1
   54: R    REDUCE
   55: b    BUILD
   56: .    STOP
highest protocol among opcodes = 2
>>> print repr(p2)
'\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.'


anselm@Emmy:~/sc/eclipsews/fg2py$ fg2python
Python 2.7.5 (default, May 18 2013, 17:02:17) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
>>> import collections
>>> N=collections.namedtuple("N","a")
>>> pickle.loads('\x80\x02c__main__\nN\nK\x01\x85\x81ccollections\nOrderedDict\n]](U\x01aK\x01ea\x85Rb.')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1419, in loads
    return Unpickler(file).load()
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 895, in load
    dispatch[key](self)
  File "/home/anselm/sc/eclipsews/fg2py/arch/rhel4u4-x86_64/lib/python2.7/pickle.py", line 1261, in load_build
    d = inst.__dict__
AttributeError: 'N' object has no attribute '__dict__'


As we can see from the trace back, the problem arises from the pickle op-code 'BUILD'. BUILD requires that the object to be build either has a method __setstate__ or has an attribute __dict__. Therefore I propose:

- Revert change 18303391b981 and add a __getstate__ method
  This is the Python 3 fix for the problem.

or

- Add a method __setstate__:

  def __setstate__(self, state): 
     """For compatibility with Python 2.7.3 and 2.7.4"""
     pass

Just for the records: the patch works as expected.

I created a small *.pth to monkey patch collections.py until 2.7.6 gets released. Maybe this is useful for someone else. Therefore I attach it here.

The pth file runs the following code during Python startup:

import collections
def _fix_issue_18015(collections):
    try:
        template = collections._class_template
    except AttributeError:
        # prior to 2.7.4 _class_template didn't exists
        return
    if not isinstance(template, basestring):
        return  # strange
    if "__dict__" in template or "__getstate__" in template:
        return  # already patched
    lines = template.splitlines()
    indent = -1
    for i,l in enumerate(lines):
        if indent < 0:
            indent = l.find('def _asdict')
            continue
        if l.startswith(' '*indent + 'def '):
            lines.insert(i, ' '*indent + 'def __getstate__(self): pass')
            lines.insert(i, ' '*indent + '__dict__ = _property(_asdict)')
            break
    collections._class_template = '''\n'''.join(lines)
_fix_issue_18015(collections)

New changeset 687295c6c8f2 by Raymond Hettinger in branch '2.7':
Issue #18015: Fix unpickling of 2.7.3 and 2.7.4 namedtuples.
http://hg.python.org/cpython/rev/687295c6c8f2