classification
Title: private dispatch table for picklers
Type: enhancement Stage: resolved
Components: Versions: Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, pitrou, python-dev, sbt
Priority: normal Keywords: patch

Created on 2012-03-01 14:34 by sbt, last changed 2012-03-04 17:39 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
pickle_dispatch.patch sbt, 2012-03-01 14:34
pickle_dispatch.patch sbt, 2012-03-04 14:51 review
Messages (6)
msg154695 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-03-01 14:34
Currently the only documented way to have customised pickling for a type is to register a reduction function with the global dispatch table managed by the copyreg module.  But such global changes are liable to disrupt other code which uses pickling.

Multiprocessing deals with this by defining a ForkingPickler class which subclasses the pure python _Pickler class (using undocumented features), and supports registering reduction functions specifically for that class.

I would like to see some documented alternative which works with both C and Python implementations.  At least then multiprocessing can avoid using slow pure python pickling.  

The attached patch allows a pickler object to have a private dispatch table which it uses *instead* of the global one.  It lets one write code like

    p = pickle.Pickler(...)
    p.dispatch_table = copyreg.dispatch_table.copy()
    p.dispatch_table[SomeClass] = reduce_SomeClass

or

    class MyPickler(pickle.Pickler):
        dispatch_table = copyreg.dispatch_table.copy()

    MyPickler.dispatch_table[SomeClass] = reduce_SomeClass
    p = MyPickler(...)

The equivalent using copyreg would be

    copyreg.pickle(SomeClass, reduce_SomeClass)
    p = pickle.Pickler(...)
msg154847 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-03 19:56
That looks like a good idea.
I don't understand the following code:

+        try:
+            self._dispatch_table = self.dispatch_table
+        except AttributeError:
+            self._dispatch_table = dispatch_table

... since self.dispatch_table is a property returning self._dispatch_table. Did you mean type(self).dispatch_table?

Also, you need to update the docs (which will also make the intended semantics of the patch clearer :-)).
msg154893 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-03-04 13:49
> I don't understand the following code:
> ... 
> since self.dispatch_table is a property returning
> self._dispatch_table. Did you mean type(self).dispatch_table?

More or less.  That code was a botched attempt to match the behaviour of the C implementation.

The C implementation does not expose the dispatch table unless it has been explicitly set (on the pickler or the pickler class), and it ignores any "dispatch_table" (or "persistent_id") attribute on the metaclass.

I will do a fixed patch with docs.
msg154897 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-03-04 14:51
Updated patch with docs.
msg154901 - (view) Author: Roundup Robot (python-dev) Date: 2012-03-04 17:35
New changeset f7a9a10ae0c0 by Antoine Pitrou in branch 'default':
Issue #14166: Pickler objects now have an optional `dispatch_table` attribute which allows to set custom per-pickler reduction functions.
http://hg.python.org/cpython/rev/f7a9a10ae0c0
msg154902 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-03-04 17:39
I've replaced occurrences of "pickle.Pickler" with "self.pickler_class" in the tests. Otherwise, the patch looks perfect and I've now committed it. Thank you!
History
Date User Action Args
2012-03-04 17:39:05pitrousetstatus: open -> closed
resolution: fixed
messages: + msg154902

stage: patch review -> resolved
2012-03-04 17:35:49python-devsetnosy: + python-dev
messages: + msg154901
2012-03-04 14:51:04sbtsetfiles: + pickle_dispatch.patch

messages: + msg154897
2012-03-04 13:49:26sbtsetmessages: + msg154893
2012-03-03 19:56:47pitrousetmessages: + msg154847
stage: patch review
2012-03-02 04:36:55eric.araujosetnosy: + pitrou, alexandre.vassalotti
2012-03-01 14:34:09sbtcreate