Hello,

while porting something to Python 3.1a1
I found out that Python 3 cannot load most Python 2 pickles
of any protocol because copy_reg has been renamed to copyreg.

Found this comment by Skip Montanaro in related issue:
http://bugs.python.org/issue3799#msg76196

Could not find an issue opened for this though.
So I'm opening one.

Regards,
Matthias Kievernagel

bpo-3675 is a similar issue, too bad nothing could be done to solve it...

If I understood correctly, bpo-3675 is about making pickle data generated
by Python 3 readable by Python 2. However, this issue is about
compatibility in the other direction—i.e., making Python 2 pickles
readable by Python 3, which is easier.

I have a patch that make Unpickler translates the old names to the new
ones at runtime. The only thing missing in the patch is the unit tests
since I am not sure this should be tested. I am thinking using the
approach Collin Winter used for his compatibility tests in bpo-5665.
However, his approach requires bidirectional pickle compatibility
between Python 2 and 3, which we don't have.

If I understood correctly, bpo-3675 is about making pickle data generated
by Python 3 readable by Python 2.

Only if a protocol <= 2 is specified. Therefore it seems it's "only" a
matter of translating module names.

Applied the patch
http://bugs.python.org/file14124/compat_pickle.diff
to rev. 73106.
Patch applies fine, 'make test' passes
and it solves my problem.
(which is far from a complete test case though

• only 5 small pickles)

Thanks,
Matthias Kievernagel

Some comments on the patch:

• I don't understand why you create a static "twotuple" object rather
  than simply using Py_BuildValue("(OO)", ...). Mutating tuples is bad.
• I don't think you need to call PyDict_Contains() before
  PyDict_GetItem(). The latter will simply return NULL if the key isn't found.
• You should check whether the item fetched from the dictionary has the
  right datatype (tuple for name_mapping, str for import_mapping). Anyone
  can change (monkeypatch) _pickle_compat and make the interpreter crash.

Unpickling e.g. StringIO objects doesn't seem to work:

>>> s = pickle.load(open("stringio.pickle", "rb"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/py3k/picklecompat-6137/Lib/pickle.py", line 1351,
in load
    encoding=encoding, errors=errors).load()
AttributeError: '_io.BytesIO' object has no attribute '__dict__'
>>>

An improved patch with tests.
It has no tests for fix_imports=False, though.

A new patch which also includes reverse mappings, so that protocol <= 2
pickles created with 3.x can also work under 2.x.
(that is, it also solves bpo-3675)

Updated patch with a couple of documentation and function prototype fixes.

Sorry, last patch had a couple of minor issues

Here is an updated patch based on Antoine's latest patch.

Summary of changes:

• Updated docstrings of Pickler and Unpickler in the pickle module.
• Fixed pickle._Pickler to consider fix_imports only for protocol < 3
• Made module name remapping in _pickle more robust:
  • added PyUnicode_Check on global_name and module_name;
  • used PyDict_GetItemWithError instead of PyDict_GetItem
• Changed Py_BuildValue("(OO)", ...) to its faster equivalent
  PyTuple_Pack(2, ...).

I don't really the idea of remapping names generated by Pickler, since
it breaks the identity guarantee in save_global(). However, I agree this
is an example where practicality beats purity. So, I do not oppose to
the change.

The latest patch is missing the file "Lib/_compat.pickle.py". (Seems as
if you forgot to svn add it after patching.)

Oops. Here is a new patch with _compat_pickle.py.

Committed in r73236 in the hope that it gets a bit more testing before
rc2/final.

I think it is worth noting that now some Python 3.1 protocol 2 pickles
can't be read by Python 3.0. We probably don't have to do anything about
that, but perhaps it should be mentioned somewhere? Docs, release notes?

I've added an entry in the what's new file in r73254.

Guido, Raymond suggested we ask your input on this one, although it's
obviously a bit late (the patch has been committed).

I'm not on IRC. What exactly is your question for me? What is Raymond's
view?

I vaguely remember you rejecting a proposal along these lines when Brett
was doing the library renaming.

The patch (as applied) turns on the renaming automatically when used
with protocol 2 (i.e. all object names are stored with their 2.x names,
not their new names). Hagen points out that 3.1 proto 2 pickles can't
be read by 3.0. I would think that the back-translation should be
disabled by default or else we're going to live with the 2.x names for a
very long time.

I don't care much one way or the other. Just thought you should see it
before it got released and set in stone. It's not too late for a one
line change, setting the fix_imports default from True to False.

FWIW, the argument for leaving the default as True is the theory that
anyone using protocol 2 is likely doing so to exchange data with 2.x.

Ah. How about only doing back-translation when protocol=2 (or lower)
is explicitly selected?

I don't much like that 3.0 will be to read pickles written by 3.1 with
the default protocol (i.e. 3), but I don't mind breaking protocol 2,
since that's most likely (as you say) intended for Python 2. So the
default for fix_imports would be None, and the __init__ would check if
it's None, and then set it to (protocol <= 2).

Ah. How about only doing back-translation when protocol=2 (or lower)
is explicitly selected?

Well, this is exactly what is implemented!

I don't much like that 3.0 will be to read pickles written by 3.1 with
the default protocol (i.e. 3),

I suppose you meant to say "will be unable", but it isn't, since
protocol 3 pickles are not affected at all by this patch.

On Tue, Jun 9, 2009 at 12:16 PM, Antoine Pitrou<report@bugs.python.org> wrote:

Antoine Pitrou <pitrou@free.fr> added the comment:

> Ah. How about only doing back-translation when protocol=2 (or lower)
> is explicitly selected?

Well, this is exactly what is implemented!

Ah, I missed that. Fine then!

> I don't much like that 3.0 will be to read pickles written by 3.1 with
> the default protocol (i.e. 3),

I suppose you meant to say "will be unable",

Indeed.

but it isn't, since
protocol 3 pickles are not affected at all by this patch.

Good.

So I'm okay with the status quo -- too bad 3.0 can't read those
pickles, but that's the deal with abandoning 3.0...

Thanks for taking a look and opining.