classification
Title: Pickling of NoneType raises PicklingError
Type: behavior Stage: resolved
Components: Documentation Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: alexandre.vassalotti Nosy List: alexandre.vassalotti, belopolsky, georg.brandl, hagen, jason.coombs, july, ncoghlan, pitrou, python-dev
Priority: normal Keywords: easy, patch

Created on 2009-07-13 17:20 by july, last changed 2013-12-01 19:44 by alexandre.vassalotti. This issue is now closed.

Files
File name Uploaded Description Edit
backport_pickle_singleton_types_fix.diff alexandre.vassalotti, 2013-12-01 02:37 review
Messages (17)
msg90496 - (view) Author: July Tikhonov (july) * Date: 2009-07-13 17:20
Python 3.2a0 (py3k:73749M, Jul  1 2009, 23:17:59)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pickle
[40072 refs]
>>> pickle.dumps(type(None))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.2/pickle.py", line 1358, in dumps
    Pickler(f, protocol, fix_imports=fix_imports).dump(obj)
_pickle.PicklingError: Can't pickle <class 'NoneType'>: attribute 
lookup builtins.NoneType failed
[40137 refs]
>>>
msg90538 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-07-15 17:09
I don't see why you want to pickle NoneType. Do you have a proper
use-case for this, or are you just playing around with pickle?
msg90540 - (view) Author: July Tikhonov (july) * Date: 2009-07-15 17:32
No, my program failed on this. It was not a big problem to manage this, 
but I think it is a bug. And it isn't documented (or I can't find it).

Other built-in types have no such problem. Is there something special 
with NoneType?
msg90555 - (view) Author: Hagen F├╝rstenau (hagen) Date: 2009-07-16 06:00
> but I think it is a bug

I think it is either a feature request (make NoneType picklable) or a
documentation issue (document that it's not).
msg90641 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2009-07-17 22:21
Given that dumps(type(Ellipsis)) and dumps(type(NotImplemented)) don't
work either, I am reclassifying this as a documentation bug.

The thing about the types of these three objects (None, Ellipsis,
NotImplemented) is that they are all designed to be singletons and they
all disallow creation of new instances of them.

>>> type(None)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NoneType' instances
>>> type(NotImplemented)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'NotImplementedType' instances
>>> type(Ellipsis)()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: cannot create 'ellipsis' instances

This does mean None has to be special cased if pickling "(object,
type(object))" pairs, but passing types through pickle strikes me as a
dubious enough exercise that it would take a fairly convincing use case
to accept an RFE to make NoneType pickleable (note that any such patch
would have to make sure that doing loads() on such a pickle didn't
manage to create a second copy of NoneType or None).
msg90648 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-07-17 22:58
I agree with Nick.

And if you really want to, you could hack a Pickler subclass to support
NoneType:

import io
import pickle

class XPickler(pickle.Pickler):
  def persistent_id(self, obj):
    if obj is type(None):
      return "NoneType"
    return None

class XUnpickler(pickle.Unpickler):
  def persistent_load(self, persistent_id):
    if persistent_id == "NoneType":
      return type(None)

def dumps(obj):
    f = io.BytesIO()
    XPickler(f).dump(obj)
    return f.getvalue()

def loads(s):
    return XUnpickler(io.BytesIO(s)).load()

Not super elegant, but it works:

>>> loads(dumps([type(None), None]))
[<type 'NoneType'>, None]
msg108957 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2010-06-29 23:19
Ellipsis and NotImplemented are different from None because they are not pickleable themselves.  The None situation is more similar to that of say module level functions:


>>> def f(): pass
... 
>>> dumps(f)  # works
b'\x80\x03c__main__\nf\nq\x00.'
>>> dumps(type(f))  # does not work
Traceback (most recent call last):
  ..
_pickle.PicklingError: Can't pickle <class 'function'>: attribute lookup builtins.function failed


I think documentation at http://docs.python.org/dev/py3k/library/pickle.html#what-can-be-pickled-and-unpickled

Can be improved in two ways:

1. s/types/objects/ in "The following types can be pickled"

2. Add a note that "type(x) can be pickled" does not follow from "x can be pickled."  The object type(x) must itself fall into one on the listed categories.

3. "built-in functions defined at the top level of a module" should probably be "built-in classes or functions defined at the top level of the builtins module"

4. "instances of such classes" should be "instances of picklable classes"

5. "__setstate__() is picklable" should be "output of __getstate__() is picklable"
msg141106 - (view) Author: Jason R. Coombs (jason.coombs) * (Python committer) Date: 2011-07-25 17:38
I've encountered a use-case where the need to pickle NoneType is more relevant (and difficult to work around).

We have a strongly-type Limited Python language, LimPy, which is based on Python (https://bitbucket.org/yougov/limpy). This parser takes, as part of its initialization arguments, a type specification (indicating which types are allowed and not allowed). In some cases, the return value may be `None`, in which case the specification says the type must be `NoneType`.

We're attempting to run this parser in a separate process, using the multiprocessing module, which requires that the arguments passed to and from the parser be pickleable. Unfortunately, because `NoneType` is in the type specification, it cannot be passed to the parser.

Here's an example of one such type specification (from the test suite):

    class SomeFunctionNamespace:
        @signature([IListType], [], None, ListOfInt)
        def listcount(self, l):
            return range(len(l))

        @signature([IListType], [], None, int)
        def listlen(self, l):
            return len(l)

        # ...

        @signature([IIntType], [IStringType], IStringType, NoneType)
        def givespec(self, *args):
            for a in args:
                print a


Note that we can pickle `str` and `int` just fine. Only type(None) fails.

It would be possible to re-write the entire LimPy system (and its child projects) to use a different object where currently NoneType is used, though NoneType is precisely the right thing to be used here except that it can't be pickled.

Since type(None) is a fundamental Python type, it strikes me as a bug that it's not pickleable, though I concede that it's also reasonable to interpret this issue as a feature request (as it's never been pickleable).

Nick makes some good comments that pickling of NoneType should be done right, but other than that, there haven't been any reasons why in principle NoneType should not be pickleable.

NoneType is more akin to `str` and `int` which are pickleable than it is to a function or type(NotImplemented), and this is evident by the way that LimPy uses it (before there was any consideration for pickleability).

Therefore, I propose we reconsider that NoneType should be made pickleable.
msg204862 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-01 00:21
New changeset 16eba94d3cfe by Alexandre Vassalotti in branch '3.3':
Issue #6477: Added support for pickling the types of built-in singletons.
http://hg.python.org/cpython/rev/16eba94d3cfe

New changeset ff56f48b3277 by Alexandre Vassalotti in branch 'default':
Issue #6477: Merge with 3.3.
http://hg.python.org/cpython/rev/ff56f48b3277
msg204866 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-01 00:57
New changeset fbb97f6eb3b3 by Alexandre Vassalotti in branch '2.7':
Issue #6477: Added pickling support for singletons and their types.
http://hg.python.org/cpython/rev/fbb97f6eb3b3
msg204867 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-01 01:05
Uh... Your commits make PyNone_Type and PyNotImplemented_Type public APIs, which I don't think is ok, especially not in bugfix releases.

Also, it would be nice to post patches on the tracker before committing (not for trivial patches of course, but it's generally nicer).
msg204869 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-12-01 01:41
Would you be okay with removing the static declaration of PyNotImplemented_Type and PyNone_Type if we prefix their name with an underscore? There isn't any other way to fix this without making the types linkable.

I might revert the 2.7 change anyway as it broke test_xpickle and it doesn't look easy to fix.
msg204870 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-01 01:44
New changeset d964d7023aa4 by Alexandre Vassalotti in branch '2.7':
Issue #6477: Revert fbb97f6eb3b3 as it broke test_xpickle.
http://hg.python.org/cpython/rev/d964d7023aa4
msg204871 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-01 01:47
Yeah, adding an underscore is a fine way of dealing with this if the symbol has to be exported.
msg204874 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-01 02:04
New changeset 9fcba15d7685 by Alexandre Vassalotti in branch '3.3':
Issue #6477: Keep PyNotImplemented_Type and PyNone_Type private.
http://hg.python.org/cpython/rev/9fcba15d7685

New changeset 7d6c27fa7f32 by Alexandre Vassalotti in branch 'default':
Issue #6477: Merge with 3.3.
http://hg.python.org/cpython/rev/7d6c27fa7f32
msg204877 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-12-01 02:37
Antoine, are you okay with applying this fix to 2.7? Or should we just mark this as a won't fix?
msg204951 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2013-12-01 19:44
I thought it over. I don't think the fix is appropriate for 2.x, as it seems closer to being an extra feature than a bug fix.

Thanks Antoine for calling me out on this one.
History
Date User Action Args
2013-12-01 19:44:48alexandre.vassalottisetstatus: open -> closed
resolution: wont fix
messages: + msg204951

stage: patch review -> resolved
2013-12-01 02:37:27alexandre.vassalottisetfiles: + backport_pickle_singleton_types_fix.diff
priority: low -> normal

versions: - Python 3.2, Python 3.3, Python 3.4
keywords: + patch
resolution: fixed -> (no value)
messages: + msg204877
stage: resolved -> patch review
2013-12-01 02:04:47python-devsetmessages: + msg204874
2013-12-01 01:47:20pitrousetmessages: + msg204871
2013-12-01 01:44:25python-devsetmessages: + msg204870
2013-12-01 01:41:27alexandre.vassalottisetmessages: + msg204869
2013-12-01 01:05:37pitrousetstatus: closed -> open
nosy: + pitrou
messages: + msg204867

2013-12-01 00:57:49alexandre.vassalottisetstatus: open -> closed
assignee: docs@python -> alexandre.vassalotti
resolution: fixed
stage: needs patch -> resolved
2013-12-01 00:57:21python-devsetmessages: + msg204866
2013-12-01 00:21:42python-devsetnosy: + python-dev
messages: + msg204862
2012-10-02 06:22:14ezio.melottisetkeywords: + easy
stage: needs patch
versions: + Python 3.3, Python 3.4, - Python 2.6, Python 3.0, Python 3.1
2011-07-25 17:38:02jason.coombssetnosy: + jason.coombs
messages: + msg141106
2010-10-29 10:07:21adminsetassignee: georg.brandl -> docs@python
2010-06-29 23:19:10belopolskysetnosy: + belopolsky
messages: + msg108957
2009-07-17 22:58:45alexandre.vassalottisetmessages: + msg90648
2009-07-17 22:21:05ncoghlansetnosy: + georg.brandl, ncoghlan
messages: + msg90641

assignee: georg.brandl
components: + Documentation, - Library (Lib)
2009-07-16 06:00:35hagensetnosy: + hagen
messages: + msg90555
2009-07-15 17:32:13julysetmessages: + msg90540
2009-07-15 17:09:05alexandre.vassalottisetpriority: low
nosy: + alexandre.vassalotti
messages: + msg90538

2009-07-13 17:20:39julycreate