classification
Title: Pickle protocol 2.0 not loading in python 3.5
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: alexandre.vassalotti, anilredshift, pitrou, python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2016-01-05 07:39 by anilredshift, last changed 2016-01-19 10:02 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
b.py anilredshift, 2016-01-05 07:39 demonstration of bug
unpickle_broken_import.patch serhiy.storchaka, 2016-01-05 09:18 review
Messages (9)
msg257524 - (view) Author: Anil Kulkarni (anilredshift) Date: 2016-01-05 07:39
Pickles created with python 3.4.X will not load with python 3.5.X if they include a collections.OrderedDict

To reproduce this issue, simply create a pickle of an OrderedDict on python 3.4.3 with protocol=2 and try to open it on 3.5. I have included a simple script to demonstrate this issue.


I believe this is related to this bug: https://bugs.python.org/issue18473

As to the real-world implications: The python package Celery uses protocol=2 by default when serializing with pickle, and thus a celery web running 3.5 cannot receive the results of a worker running 3.4
For celery specifically, there is a workaround by setting the PICKLE_PROTOCOL environment variable, but this is a core python issue.

P.S. This is the first bug I've filed with python so please let me know if there's something else I should be including.

Thanks!
msg257526 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-05 09:18
Thank you for your report Anil.

Python 3.4.3 creates incorrect pickle with protocol 2 (it can't be load in Python 2). issue18473 fixed pickling with protocol 2 in Python 3, but broke loading broken pickles created in unpatched versions of Python 3.

Here is a patch that allows to load such broken pickles.

But it is too late to fix Python 3.4, it can get only security fixes. You have Celery specific workaround. More general workaround is to update _compact_pickle.IMPORT_MAPPING, e.g.:

import _compact_pickle
_compact_pickle.IMPORT_MAPPING.update({
    'UserDict': 'collections',
    'UserList': 'collections',
    'UserString': 'collections',
    'whichdb': 'dbm',
    'StringIO':  'io',
    'cStringIO': 'io',
})

Note that you have to set mapping not just for one module name like 'UserList', but for all 'UserDict', 'UserList', and 'UserString', because it is not predicable to what module name 'collections' is mapped in Python 3.4.3.
msg257541 - (view) Author: Anil Kulkarni (anilredshift) Date: 2016-01-05 19:53
Hi Serhiy,

I have done some more investigation this morning and I have come across two distinct issues.

The first is that pickles (specifically the pickle in my earlier message) created in python 3.0->3.4.3 do not load on python 2.7.10. The exception is 3.5.1, which produces a pickle that DOES load on 2.7.10
This is unfortunate but not a regression

The second issue is that pickles created by python 3.0->3.4.3 do NOT load in python 3.5

This is a regression, in the sense that the pickle created by 3.0->3.4.3 is compatible with every other version of python (I haven't tested every single combination here, but several)

From the language perspective, it may be that 3.4.X has incorrect code, but from a compatibility perspective, 3.5 is breaking the promise that pickles are compatible across versions of python.

I write this as justification that python 3.5 should fix this regression in compatibility with 3.0 -> 3.4.3. At the very least 3.5 should have a shim behavior to fallback and allow it to import these pickles correctly.

Thanks,
Anil
msg257557 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-05 21:23
I agree, and the provided is purposed to fix this issue. In the previous message I just proposed a workaround for those who can't wait 3.5.2.
msg257558 - (view) Author: Anil Kulkarni (anilredshift) Date: 2016-01-05 21:57
Ah, sorry I misunderstood. Thanks for the quick turnaround!
msg258540 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-18 19:36
New changeset 270add17f203 by Serhiy Storchaka in branch '3.5':
Issue #26013: Added compatibility with broken protocol 2 pickles created
https://hg.python.org/cpython/rev/270add17f203

New changeset 35ff0976b211 by Serhiy Storchaka in branch 'default':
Issue #26013: Added compatibility with broken protocol 2 pickles created
https://hg.python.org/cpython/rev/35ff0976b211
msg258544 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-01-18 20:21
Buildbots are unhappy.

http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/344/steps/test/logs/stdio

======================================================================
FAIL: test_reverse_import_mapping (test.test_pickle.CompatPickleTests) [('cStringIO', 'io')]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_pickle.py", line 353, in test_reverse_import_mapping
    (module3, module2))
AssertionError: No reverse mapping from 'io' to 'cStringIO'
msg258545 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-18 20:34
New changeset 7215d13baa2d by Serhiy Storchaka in branch '3.5':
Added exceptins for testing non-reversible import mapping for Issue #26013.
https://hg.python.org/cpython/rev/7215d13baa2d

New changeset 16cfc1652844 by Serhiy Storchaka in branch 'default':
Added exceptins for testing non-reversible import mapping for Issue #26013.
https://hg.python.org/cpython/rev/16cfc1652844
msg258546 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-18 20:46
Thank you Victor.
History
Date User Action Args
2016-01-19 10:02:59serhiy.storchakasetstatus: open -> closed
resolution: fixed
2016-01-18 20:46:05serhiy.storchakasetmessages: + msg258546
2016-01-18 20:34:43python-devsetmessages: + msg258545
2016-01-18 20:21:54vstinnersetstatus: closed -> open

nosy: + vstinner
messages: + msg258544

resolution: fixed -> (no value)
2016-01-18 19:41:25serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-01-18 19:36:33python-devsetnosy: + python-dev
messages: + msg258540
2016-01-05 21:57:42anilredshiftsetmessages: + msg257558
2016-01-05 21:23:59serhiy.storchakasetmessages: + msg257557
2016-01-05 19:53:28anilredshiftsetmessages: + msg257541
2016-01-05 09:18:35serhiy.storchakasetfiles: + unpickle_broken_import.patch

assignee: serhiy.storchaka
components: + Library (Lib)
versions: + Python 3.6
keywords: + patch
nosy: + alexandre.vassalotti, serhiy.storchaka, pitrou

messages: + msg257526
stage: patch review
2016-01-05 07:39:44anilredshiftcreate