classification
Title: __newobj__ pickle feature is not documented
Type: Stage:
Components: Documentation Versions: Python 3.0, Python 2.6, Python 2.5
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, christian.heimes, georg.brandl
Priority: low Keywords: easy

Created on 2008-09-09 12:13 by christian.heimes, last changed 2009-06-08 21:20 by alexandre.vassalotti. This issue is now closed.

Messages (6)
msg72850 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-09-09 12:13
The pickle system has an undocumented but very useful feature. When the
first element of the tuple returned by __reduce__ is a function named
__newobj__, a special obcode is generated. __newobj__ doesn't need to be
registered as safe for unpickling, too.

From pickle.py:

            # use the more efficient NEWOBJ opcode, while still
            # allowing protocol 0 and 1 to work normally.  For this to
            # work, the function returned by __reduce__ should be
            # called __newobj__, and its first argument should be a
            # new-style class.  The implementation for __newobj__
            # should be as follows, although pickle has no way to
            # verify this:
            #
            # def __newobj__(cls, *args):
            #     return cls.__new__(cls, *args)
            #
            # Protocols 0 and 1 will pickle a reference to __newobj__,
            # while protocol 2 (and above) will pickle a reference to
            # cls, the remaining args tuple, and the NEWOBJ code,
            # which calls cls.__new__(cls, *args) at unpickling time
            # (see load_newobj below).  If __reduce__ returns a
            # three-tuple, the state from the third tuple item will be
            # pickled regardless of the protocol, calling __setstate__
            # at unpickling time (see load_build below).
            #
            # Note that no standard __newobj__ implementation exists;
            # you have to provide your own.  This is to enforce
            # compatibility with Python 2.2 (pickles written using
            # protocol 0 or 1 in Python 2.3 should be unpicklable by
            # Python 2.2).
msg75334 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-10-29 23:18
Could explain me how this feature could be used, other than for
providing the efficient and backward-compatible pickling mechanism for
new-style classes?
msg75335 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-29 23:26
Alexandre Vassalotti wrote:
> Alexandre Vassalotti <alexandre@peadrop.com> added the comment:
> 
> Could explain me how this feature could be used, other than for
> providing the efficient and backward-compatible pickling mechanism for
> new-style classes?

The feature makes it easy to write __reduce__ methods for subclasses of 
builtins. Take this example:

def __newobj__(cls, *args):
     return cls.__new__(cls, *args)

class mydict(dict):
     def __reduce__(self):
         state = (dict(self), self.__dict__)
         return (__newobj__, (self.__class__,), state)

Without the __reduce__ method the information in __dict__ and the class 
would be lost.

Christian
msg75336 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-10-29 23:40
Christian Heimes wrote:
> Christian Heimes <lists@cheimes.de> added the comment:
> 
> Alexandre Vassalotti wrote:
>> Alexandre Vassalotti <alexandre@peadrop.com> added the comment:
>>
>> Could explain me how this feature could be used, other than for
>> providing the efficient and backward-compatible pickling mechanism for
>> new-style classes?
> 
> The feature makes it easy to write __reduce__ methods for subclasses of 
> builtins. Take this example:
> 
> def __newobj__(cls, *args):
>      return cls.__new__(cls, *args)
> 
> class mydict(dict):
>      def __reduce__(self):
>          state = (dict(self), self.__dict__)
>          return (__newobj__, (self.__class__,), state)

Of course one has to provide a __setstate__ method, too.

     def __setstate__(self, state):
         dict.update(self, state[0])
         self.__dict__.update(state[1])
msg75352 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-10-30 03:00
> Without the __reduce__ method the information in __dict__ and 
> the class would be lost.

Are you sure about that?


Python 2.5.2 (r252:60911, Jul 31 2008, 17:28:52) 
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
>>> class mydict(dict): pass
... 
>>> D = mydict({1:"one",2:"two"})
>>> D.foo = 3
>>> import pickle
>>> E = pickle.loads(pickle.dumps(D))
>>> E.foo
3
>>> E
{1: 'one', 2: 'two'}
>>> type(E)
<class '__main__.mydict'>
>>> F = pickle.loads(pickle.dumps(D, 2))
>>> F.foo
3
>>> F
{1: 'one', 2: 'two'}
>>> type(F)
<class '__main__.mydict'>
msg89115 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-08 21:20
Closing as the feature is documented in the section "The __newobj__
unpickling function" of PEP 307.
History
Date User Action Args
2009-06-08 21:20:50alexandre.vassalottisetstatus: open -> closed
assignee: georg.brandl ->
resolution: not a bug
messages: + msg89115
2008-10-30 03:00:31alexandre.vassalottisetmessages: + msg75352
2008-10-29 23:40:00christian.heimessetmessages: + msg75336
2008-10-29 23:26:02christian.heimessetmessages: + msg75335
2008-10-29 23:18:09alexandre.vassalottisetnosy: + alexandre.vassalotti
messages: + msg75334
2008-09-09 12:13:13christian.heimescreate