I think REVERSE_IMPORT_MAPPING is wrong (see http://hg.python.org/cpython/file/7272ef213b7c/Lib/_compat_pickle.py#l80 ). It relies on dictionary items order and maps 'collections' module to either UserString or UserList. This makes it impossible to unpickle collections module classes pickled by Python 3.x when using Python 2.x.

>>> import collections
>>> dct = collections.defaultdict()
>>> pickle.dumps(dct, protocol=1)
b'cUserString\ndefaultdict\nq\x00)Rq\x01.'
>>> pickle.dumps(dct, protocol=2, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'

I think pickling of instances of classes from other modules could also suffer. I don't think it is a good idea to map io to cStringIO or StringIO, or to map http.server to either BaseHTTPServer, SimpleHTTPServer or CGIHTTPServer.

Originally found this issue here: https://github.com/nltk/nltk/pull/441/files#L2R67

my copy-paste went wrong, last statement in the example should be

>>> pickle.dumps(dct, protocol=1, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'

This patch only addresses the proven errors with UserList, UserString, and collections.

I think that both UserDict.UserDict and UserDict.IterableUserDict should be mapped to collections.UserDict. And reverse mapping should map collections.UserDict to UserDict.IterableUserDict.

There are similar issues with other "multiple to single" mappings (e.g. to "io", "dbm", "http.server").

Here is a patch which fixes multiple reversible mappings.

Mappings for StringIO and cStringIO is removed at all because cStringIO.StringIO is not pickleable on 2.7 at all and pickled StringIO.StringIO is not compatible with 3.x (bpo-22890).

Ping.

Updated patch fixes more bugs.

Added support of unicode, izip_longest, abstract collections, ChainMap, multiprocessing exceptions, some socket and multiprocessing functions and types, xml.etree.ElementTree (C implementation). Added support of urllib and urllib2 (only public interface). Fixed reverse mapping of OSError subclasses, str, bz2, different dbm implementations, pickle. Added special sanity tests for _compat_pickle mappings (test that mappings are not ambiguous and mainly reversible, that 3.x names exist, etc).

New changeset 8d86dfe53b97 by Serhiy Storchaka in branch '3.4':
Issue bpo-18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/8d86dfe53b97

New changeset 5980e81219ed by Serhiy Storchaka in branch 'default':
Issue bpo-18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/5980e81219ed

Buildbots are unhappy. Example:

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11437/steps/test/logs/stdio

======================================================================
ERROR: test_name_mapping (test.test_pickle.CompatPickleTests) [(('tkinter.filedialog', 'FileDialog'), ('FileDialog', 'FileDialog'))]
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'tkinter.filedialog'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 286, in test_name_mapping
    attr = getattribute(module3, name3)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 247, in getattribute
    obj = getmodule(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 243, in getmodule
    __import__(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/tkinter/__init__.py", line 38, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named '_tkinter'

A different error:

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2880/steps/test/logs/stdio

======================================================================
FAIL: test_class_nested_enum_and_pickle_protocol_four (test.test_enum.TestEnum)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 580, in test_class_nested_enum_and_pickle_protocol_four
    protocol=(0, 3))
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 82, in test_pickle_exception
    dumps(obj, protocol=protocol)
AssertionError: PicklingError not raised

A third kind of error:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/5992/steps/test/logs/stdio

======================================================================
ERROR: test_import (test.test_pickle.CompatPickleTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'multiprocessing.popen_fork'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 266, in test_import
    getmodule(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 243, in getmodule
    __import__(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 13, in <module>
    class Popen(object):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 25, in Popen
    def poll(self, flag=os.WNOHANG):
AttributeError: module 'os' has no attribute 'WNOHANG'

New changeset 5c5eb374e296 by Serhiy Storchaka in branch '3.4':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/5c5eb374e296

New changeset 29b2b2d8e36f by Serhiy Storchaka in branch 'default':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/29b2b2d8e36f

Fixed import errors. A second kind of errors was related to bpo-23611.

Le mardi 31 mars 2015, Roundup Robot <report@bugs.python.org> a écrit :

Roundup Robot added the comment:

New changeset 5c5eb374e296 by Serhiy Storchaka in branch '3.4':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/5c5eb374e296

New changeset 29b2b2d8e36f by Serhiy Storchaka in branch 'default':
Issue bpo-18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/29b2b2d8e36f

Why do you catch AttributeError on import?

Why do you catch AttributeError on import?

Because importing multiprocessing.popen_fork raises AttributeError on Windows.