classification
Title: some objects pickled by Python 3.x are not unpicklable in Python 2.x because of incorrect REVERSE_IMPORT_MAPPING
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: alexandre.vassalotti, doug.royal, kmike, pitrou, python-dev, serhiy.storchaka, vstinner
Priority: high Keywords: needs review, patch

Created on 2013-07-16 11:22 by kmike, last changed 2015-03-31 17:23 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
fix_issue18473.patch doug.royal, 2014-11-07 20:38 review
pickle_fix_import.patch serhiy.storchaka, 2014-11-17 07:47 review
pickle_fix_import_2.patch serhiy.storchaka, 2015-03-24 15:38 review
Messages (15)
msg193160 - (view) Author: Mikhail Korobov (kmike) Date: 2013-07-16 11:22
I think REVERSE_IMPORT_MAPPING is wrong (see http://hg.python.org/cpython/file/7272ef213b7c/Lib/_compat_pickle.py#l80 ). It relies on dictionary items order and maps 'collections' module to either UserString or UserList. This makes it impossible to unpickle collections module classes pickled by Python 3.x when using Python 2.x.

>>> import collections
>>> dct = collections.defaultdict()
>>> pickle.dumps(dct, protocol=1)
b'cUserString\ndefaultdict\nq\x00)Rq\x01.'
>>> pickle.dumps(dct, protocol=2, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'

I think pickling of instances of classes from other modules could also suffer. I don't think it is a good idea to map io to cStringIO or StringIO, or to map http.server to either BaseHTTPServer, SimpleHTTPServer or CGIHTTPServer.

Originally found this issue here: https://github.com/nltk/nltk/pull/441/files#L2R67
msg193161 - (view) Author: Mikhail Korobov (kmike) Date: 2013-07-16 11:24
my copy-paste went wrong, last statement in the example should be

>>> pickle.dumps(dct, protocol=1, fix_imports=False)
b'ccollections\ndefaultdict\nq\x00)Rq\x01.'
msg230828 - (view) Author: Doug Royal (doug.royal) * Date: 2014-11-07 20:38
This patch only addresses the proven errors with UserList, UserString, and collections.
msg231021 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-11 10:39
I think that both UserDict.UserDict and UserDict.IterableUserDict should be mapped to collections.UserDict. And reverse mapping should map collections.UserDict to UserDict.IterableUserDict.

There are similar issues with other "multiple to single" mappings (e.g. to "io", "dbm", "http.server").
msg231269 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-17 07:47
Here is a patch which fixes multiple reversible mappings.

Mappings for StringIO and cStringIO is removed at all because cStringIO.StringIO is not pickleable on 2.7 at all and pickled StringIO.StringIO is not compatible with 3.x (issue22890).
msg232659 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-12-15 12:35
Ping.
msg239145 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-24 15:38
Updated patch fixes more bugs.

Added support of unicode, izip_longest, abstract collections, ChainMap, multiprocessing exceptions, some socket and multiprocessing functions and types, xml.etree.ElementTree (C implementation). Added support of urllib and urllib2 (only public interface). Fixed reverse mapping of OSError subclasses, str, bz2, different dbm implementations, pickle. Added special sanity tests for _compat_pickle mappings (test that mappings are not ambiguous and mainly reversible, that 3.x names exist, etc).
msg239678 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 10:19
New changeset 8d86dfe53b97 by Serhiy Storchaka in branch '3.4':
Issue #18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/8d86dfe53b97

New changeset 5980e81219ed by Serhiy Storchaka in branch 'default':
Issue #18473: Fixed 2to3 and 3to2 compatible pickle mappings.
https://hg.python.org/cpython/rev/5980e81219ed
msg239685 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-31 11:54
Buildbots are unhappy. Example:

http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/11437/steps/test/logs/stdio


======================================================================
ERROR: test_name_mapping (test.test_pickle.CompatPickleTests) [(('tkinter.filedialog', 'FileDialog'), ('FileDialog', 'FileDialog'))]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'tkinter.filedialog'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 286, in test_name_mapping
    attr = getattribute(module3, name3)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 247, in getattribute
    obj = getmodule(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_pickle.py", line 243, in getmodule
    __import__(module)
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/tkinter/__init__.py", line 38, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named '_tkinter'
msg239686 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-31 12:00
A different error:

http://buildbot.python.org/all/builders/AMD64%20Snow%20Leop%203.x/builds/2880/steps/test/logs/stdio

======================================================================
FAIL: test_class_nested_enum_and_pickle_protocol_four (test.test_enum.TestEnum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 580, in test_class_nested_enum_and_pickle_protocol_four
    protocol=(0, 3))
  File "/Users/buildbot/buildarea/3.x.murray-snowleopard/build/Lib/test/test_enum.py", line 82, in test_pickle_exception
    dumps(obj, protocol=protocol)
AssertionError: PicklingError not raised
msg239687 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-31 12:00
A third kind of error:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/5992/steps/test/logs/stdio

======================================================================
ERROR: test_import (test.test_pickle.CompatPickleTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 241, in getmodule
    return sys.modules[module]
KeyError: 'multiprocessing.popen_fork'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 266, in test_import
    getmodule(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\test\test_pickle.py", line 243, in getmodule
    __import__(module)
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 13, in <module>
    class Popen(object):
  File "C:\buildbot.python.org\3.x.kloth-win64\build\lib\multiprocessing\popen_fork.py", line 25, in Popen
    def poll(self, flag=os.WNOHANG):
AttributeError: module 'os' has no attribute 'WNOHANG'
msg239712 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-31 13:58
New changeset 5c5eb374e296 by Serhiy Storchaka in branch '3.4':
Issue #18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/5c5eb374e296

New changeset 29b2b2d8e36f by Serhiy Storchaka in branch 'default':
Issue #18473: Fixed pickle compatibility tests for optional modules.
https://hg.python.org/cpython/rev/29b2b2d8e36f
msg239713 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-31 14:00
Fixed import errors. A second kind of errors was related to issue23611.
msg239735 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-31 17:10
Le mardi 31 mars 2015, Roundup Robot <report@bugs.python.org> a écrit :

>
> Roundup Robot added the comment:
>
> New changeset 5c5eb374e296 by Serhiy Storchaka in branch '3.4':
> Issue #18473: Fixed pickle compatibility tests for optional modules.
> https://hg.python.org/cpython/rev/5c5eb374e296
>
> New changeset 29b2b2d8e36f by Serhiy Storchaka in branch 'default':
> Issue #18473: Fixed pickle compatibility tests for optional modules.
> https://hg.python.org/cpython/rev/29b2b2d8e36f
>

Why do you catch AttributeError on import?
msg239739 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-31 17:23
> Why do you catch AttributeError on import?

Because importing multiprocessing.popen_fork raises AttributeError on Windows.
History
Date User Action Args
2015-03-31 17:23:32serhiy.storchakasetmessages: + msg239739
2015-03-31 17:10:37vstinnersetmessages: + msg239735
2015-03-31 16:48:43serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-03-31 14:00:33serhiy.storchakasetmessages: + msg239713
2015-03-31 13:58:41python-devsetmessages: + msg239712
2015-03-31 12:00:56vstinnersetmessages: + msg239687
2015-03-31 12:00:24vstinnersetmessages: + msg239686
2015-03-31 11:54:52vstinnersetnosy: + vstinner
messages: + msg239685
2015-03-31 10:19:18python-devsetnosy: + python-dev
messages: + msg239678
2015-03-24 15:38:05serhiy.storchakasetfiles: + pickle_fix_import_2.patch

messages: + msg239145
2014-12-15 12:35:17serhiy.storchakasetkeywords: + needs review

messages: + msg232659
2014-11-17 07:47:49serhiy.storchakasetstage: needs patch -> patch review
2014-11-17 07:47:20serhiy.storchakasetfiles: + pickle_fix_import.patch

messages: + msg231269
2014-11-11 10:39:13serhiy.storchakasetnosy: + serhiy.storchaka

messages: + msg231021
versions: + Python 3.5, - Python 3.3
2014-11-07 20:38:49doug.royalsetfiles: + fix_issue18473.patch

nosy: + doug.royal
messages: + msg230828

keywords: + patch
2013-12-01 21:29:14alexandre.vassalottisetpriority: normal -> high
assignee: alexandre.vassalotti
stage: needs patch
type: behavior
versions: + Python 3.4
2013-07-19 21:56:34terry.reedysetnosy: + pitrou, alexandre.vassalotti
2013-07-16 11:24:42kmikesetmessages: + msg193161
2013-07-16 11:22:47kmikecreate