Pickling exceptions fails when an Exception class
requires an argument in the constructor, but doesn't
call its base class' constructor.  See this mail
for details:

http://mail.python.org/pipermail/python-dev/2007-April/072416.html

This patch simply moves initial args assignment to
BaseException.__new__. This should fix most of the
problems, because it is very unlikely that an
exception overwrites the __new__ method; exceptions
used to be old style classes, which don't support
the __new__ special method.

The args attribute is still overwritten in all the
__init__ methods, so there shouldn't be any backward
compatibility problems.

File Added: exc_args_25.diff

I'm attaching a test that Eric Huss sent in private
mail.

The test fails, because old exception pickles don't
have args for the reconstructor, but their __init__()
gets called anyway because they are new style classes
now.

The problem is in cPickle.InstanceNew() and
pickle.Unpickler._instantiate().  Those methods behave
differently depending on the type of the object
instantiated; they avoid the __init__() call when type
is an old style class.

There is nothing that can be done in the exception
classes to fix this issue; the fix would need to be
in the pickle and cPickle module.
File Added: test_exception_pickle.py

I have stumbled across another scenario where unpickling fails.  If your exception takes arguments, but you call Exception.__init__ with a different number of arguments, it will fail.  As in:

class D(Exception):
    def __init__(self, foo):
        self.foo = foo
        Exception.__init__(self)

Added patch #1744398 as an alternate solution.

Attaching a new patch that fixes the 

class D(Exception):
    def __init__(self, foo):
        self.foo = foo
        Exception.__init__(self)

scenario by keeping the original args to __new__ as an exception attribute.
File Added: exception-pickling.diff

self->args needs to be initialized to NULL because if the alloc of args failed, self->args would be uninitialized and deallocated.  (I realize the alloc of an empty tuple will realistically never fail currently.)

I'm not sure if this could cause any new problems because of the behavior change, but the code itself looked fine to me.  Hopefully someone with more knowledge of exceptions could take a look at the idea.

Georg's new patch looks good to me, it seems to pass all the scenarios that I know of.  You can close out my patch if you check this in.  Thanks for looking into this!

The only question left is what to do if something reassigns exceptioninstance.args.

There is also the problem that Jim Fulton mentioned
in bug #1742889; if the user modifies the 'message'
attribute of an exception (or any other attribute
that is not stored in the __dict__), it won't get
serialized on pickling.

Attaching a new patch that fixes this by using the
__getstate__() pickling hook.  It uses tp_members
and tp_getset members of the type object to find
all the attributes that need to be serialized.

The 2.5 patch also contains a fix for migrating
old exception pickles to Python 2.5.  For that
I had to change cPickle and pickle.  It would
be nice if Jim could comment if this change is
needed, since ZODB is probably the biggest user
of these modules.

The downside is that this patch is fairly big.

File Added: exception_pickling_25.diff

File Added: exception_pickling_26.diff

I wouldn't care too much about .message; it was new in 2.5 and is already deprecated in 2.6.

It's not just the message attribute; the problem is
that the current code expects that the exception
won't be modified between creation and unpickling.

For example:

>>> import pickle
>>> e = EnvironmentError()
>>> e.filename = "x"
>>> new = pickle.dumps(pickle.loads(e))
>>> print new.filename
None
>>> e = SyntaxError()
>>> e.lineno = 10
>>> new = pickle.loads(pickle.dumps(e))
>>> print new.lineno
None

Yes, this is what I meant with my last comment.

Raising priority.

It's not clear to me what the next step is here, does one or more of the
other folks need to provide some input?  Georg: If you are looking for
something from someone, can you assign the ticket to them?

IMO zseil's latest patch is the best one, but I need any other developer
to review and approve it.

While zseil's patch for Python 2.5 works for me (on the current 2.5.1
download), svn version of Python 2.6 rejects the 2.6 patch. Attaching
fixed 2.6 patch (2 rejects, 1 fuzz fixed, patch works without complains
for me). I would be very glad if someone could review the patches and
maybe commit them.

The last patch (exception_pickling_26.diff) applies cleanly, and all the
tests run ok.

System:
  Python 2.6a0 (trunk:60073M, Jan 19 2008, 11:41:33) 
  [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2

I'm against including this in 2.5.2. It reeks of a new feature, and the
code looks like it would risk breaking other apps.

(I'm fine with adding this to 2.6 of course.)

To me it seems more like a bug fix - in Python 2.4 and older the
pickling works well and the current 2.5 behavior really breaks existing
code. That's why I plead for inclusion in 2.5.2; because this issue
prevents our company to move to 2.5.

Understood.  Can you get other developers on python-dev to weigh in? 
Maybe I am over-estimating the danger.

I tested exception_pickling_25.diff, and it may break existing code.
In 2.5.1, Exception("Hello,4).__reduce__() gives 
(<type 'exceptions.Exception'>, ('Hello', 4))

With the patch, it gives

TypeError: can't pickle Exception objects

IMO, that is an unacceptable change for a bugfix release.

Aside: please give unique file names to the patches, or remove patches
if you want to replace a previous patch.

How is this coming? Can we apply this to 2.6?

Sorry for the long silence. I think that this patch is not relevant
anymore. The code that uses exception pickling already had to be adapted
to changes in Python 2.5, so there is no need to change the pickling
again and risk breaking user code. I'll close it in a few days if nobody
objects.

I'm disappointed to see this closed.  Exception pickling is still broken
in some cases in 2.6.

OK, reopening. Can you post an example that fails today?

In the attached test_exception_pickle.py file, class C and D cannot be
unpickled (raises TypeError).

class C(Exception):
    """Extension with values, args not set."""
    def __init__(self, foo):
        self.foo = foo

class D(Exception):
    """Extension with values, init called with no args."""
    def __init__(self, foo):
        self.foo = foo
        Exception.__init__(self)

There are also some other exceptions that fail to be handled properly. 
See the exception_pickling_26.diff for some unittests.  In particular, I
see SlotedNaiveException failing.

Ping?  Is anyone working on this?  Have Eric's concerns been addressed and can we pickle/unpickle all exceptions yet?

The attached patch adds Mark's examples to test_pickle as a failing test.

The case in msg76028 are probably not pointing to a bug.  If subclass' __init__ passes its args to the base class __init__ (as it probably should), pickling works: 

class E(Exception):
    """Extension with values, init called with args."""
    def __init__(self, foo):
        self.foo = foo
        Exception.__init__(self, foo)

>>> from pickle import *
>>> loads(dumps(E(1)))
E(1,)
>>> _.foo
1

What would be a use case for not setting args?  Even if there was a valid use case for it, it would be easy to simply provide custom __getinitargs__ or __reduce__ to support it.

Note that with msg76028 definitions, eval(repr(C('foo'))) also raises a TypeError, so properly supporting args hiding in exception subclasses would probably involve overriding __repr__ as well.

Alexander, the use case I was involved with was an RPC system which allowed exceptions to propagate over the connection.  In this case, you do not have absolute control over which exceptions may be raised, or who wrote the code that is raising the exception.  There are many cases, even in the standard library, where people do not pass all arguments to the base exception class.  Some of these are probably mistakes, but in general pickle shouldn't fail on otherwise legitimate objects.

There are other cases where passing all arguments up is not wanted, or at least cumbersome.  If you have multiple levels of inheritance, and subclasses add additional arguments to the init, they wouldn't have a way to include those arguments to the base class unless all classes were written with *args in the init function, which many people would not know or remember to do.

In msg108954, I believe belopolsky is mistaken in stating that "it would be easy to simply provide custom __getinitargs__ or __reduce__ to support it". It appears __getinitargs__ does not work on Python 2.5 or Python 2.7. Exceptions of the following class still raise a TypeError on unpickling:

class D(Exception):
    """Extension with values, init called with no args."""
    def __init__(self, foo):
        self.foo = foo
        Exception.__init__(self)

    def __getinitargs__(self):
        return self.foo,

Using __reduce__ does seem to work. I suspect this is because Exceptions are extension types.

I think the fundamental problem is that pickling exceptions does not follow the principle of least surprise. In particular:

 - Other built-in objects (dicts, lists, etc) don't require special handling (replace Exception with dict in the above example and it works).
 - It's not obvious how to write an Exception subclass that takes additional arguments and make it pickleable.
 - Neither the pickle documentation nor the Exception documentation describe how pickling is implemented in Exceptions.

Eric has provided some good use cases. Furthermore, it seems counter-intuitive to me to pass a subclass' custom arguments to the parent class. Indeed, even the official tutorial defines exception classes that are unpickleable (http://docs.python.org/tutorial/errors.html#tut-userexceptions).

If the use case is obvious enough that it shows up in the hello world tutorial, I don't think there should be any argument that it's not a common use case.

At the very least, there should be a section in the pickle documentation or Exception documentation describing how one should make a pickleable subclass. What would be better is for Exceptions to behave like most other classes when being pickled.

After some further reading, I found that PEP-352 (http://www.python.org/dev/peps/pep-0352/) explicitly states that "Including programmatic information (e.g., an error code number) should be stored as a separate attribute in a subclass [and not in the args attribute]." The parameter to Exception.__init__ should always be a single, human-readable string. The default pickler doesn't handle this officially-prescribed use-case without overriding __reduce__.

class F(Exception):
    def __init__(self, message, code):
        Exception.__init__(self, message)
        self.code = code

Of course, as belopolsky observed, __repr__ must also be overridden.

On Wed, Sep 22, 2010 at 9:39 AM, Jason R. Coombs <report@bugs.python.org> wrote:
> .. It appears __getinitargs__ does not work on Python 2.5 or Python 2.7.

Yes,  __getinitargs__  is only used for old style classes.  I was
wrong on that point.

> Exceptions of the following class still raise a TypeError on unpickling:
>
> class D(Exception):
>    """Extension with values, init called with no args."""
>    def __init__(self, foo):
>        self.foo = foo
>        Exception.__init__(self)
>
>    def __getinitargs__(self):
>        return self.foo,
>

The problem with your D class is that it does not provide correct args
attribute which is expected from an exception class:

()

> Using __reduce__ does seem to work. I suspect this is because Exceptions are extension types.
>

There are two ways to fix this class.  Both fix the args issue as well:

1. Pass foo to Exception.__init__  like this: Exception.__init__(self,
foo) in D.__init__.

2. Explicitly initialize self.args: self.args	= foo,

> I think the fundamental problem is that pickling exceptions does not follow the principle of least surprise. In particular:
>
>  - Other built-in objects (dicts, lists, etc) don't require special handling (replace Exception with dict in the above example and it works).

Other built-in objects don't provide an API for retrieving their init arguments.

>  - It's not obvious how to write an Exception subclass that takes additional arguments and make it pickleable.

AFAICT, this is python subclassing 101: if base class __init__ uses
arguments, they should be passed to it by subclass' __init__.

>  - Neither the pickle documentation nor the Exception documentation describe how pickling is implemented in Exceptions.
>

Exception documentation may be improved by adding a section on
subclassing.  Note that the args argument was not even mentioned in
2.7 documentation, so some discussion of its role may be helpful
somewhere.

> Eric has provided some good use cases. Furthermore, it seems counter-intuitive to me to pass a subclass' custom
> arguments to the parent class. Indeed, even the official tutorial defines exception classes that are unpickleable
> (http://docs.python.org/tutorial/errors.html#tut-userexceptions).
>

Well, the tutorial examples should probably be changed.  In these
examples base class __init__ is not called at all which is probably
not a good style.

> If the use case is obvious enough that it shows up in the hello world tutorial, I don't think
> there should be any argument that it's not a common use case.
>

I am not arguing against simplifying Exception subclassing.  I just
don't see an easy solution that would not promote subclasses with
unusable args attribute.  I also disagree with this issue classified
as a bug.  It may be a valid feature request, but not a bug.

In any case, no proponent of this feature has come up with a patch for
3.2 so far and in my view, this would be a prerequisite for moving
this forward.

> At the very least, there should be a section in the pickle documentation or Exception
> documentation describing how one should make a pickleable subclass. ..

I agree, but again someone has to step in to write such section.
Improving documentation may also be the only solution for the 2.x
series.

Perhaps this should be addressed separately, but subprocess.CalledProcessError is subject to this problem (can't be unpickled) (it has separate returncode and cmd attributes, but no args).

It's straightforward to conform user-defined Exceptions to including .args and having reasonable __init__ functions, but not possible in the case of stdlib exceptions.

>>> import subprocess, pickle
>>> try:
...   subprocess.check_call('/bin/false')
... except Exception as e:
...   pickle.loads(pickle.dumps(e))
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/lib/python3.1/subprocess.py", line 435, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/bin/false' returned non-zero exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/usr/lib/python3.1/pickle.py", line 1363, in loads
    encoding=encoding, errors=errors).load()
TypeError: __init__() takes at least 3 positional arguments (1 given)

What is the status on this? It contains to be an
issue. See http://bugs.python.org/issue13751 and
http://bugs.python.org/issue13760

I'd just like to weigh in and say that this is a major issue for me at the moment. Not being able to indiscriminately pickle/unpickle exceptions is making my parallel-processing work very painful, because of problematic stdlib exceptions.

I'm surprised this hasn't been fixed way back in 2.x. FWIW, for my project having this fixed in 3.x could be a significant incentive to finally ditch 2.x.