classification
Title: multiprocessing.pool hangs if any worker raises an Exception whose constructor requires a parameter
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: alexandre.vassalotti, fmitha, jnoller, pitrou, sbt, terry.reedy
Priority: normal Keywords:

Created on 2012-01-09 20:23 by fmitha, last changed 2012-05-25 20:05 by sbt. This issue is now closed.

Messages (6)
msg150975 - (view) Author: Faheem Mitha (fmitha) Date: 2012-01-09 20:23
See my question at http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing

I can't improve on the analysis by Lorenzo Bolla,
so I reproduce his example below. This example hangs if
BadExc is thrown, but not if GoodExc is thrown.
The only difference between these is that the GoodExc does
not require an argument be passed to its constructor, while
BadExc does.

This looks like a bug to me, though I suppose it might
be some pickling limitation.

I have confirmed this behavior is present in 2.6.6, 2.7.2,
and 3.1.3, all tested on Debian squeeze.

                                       Regards, Faheem

#################################################

import multiprocessing

class BadExc(Exception):
    def __init__(self, a):
        '''Non-optional param in the constructor.'''
        self.a = a

class GoodExc(Exception):
    def __init__(self, a=None):
        '''Optional param in the constructor.'''
        self.a = a

def do(kwargs):
    i = kwargs['i']
    print i
    raise BadExc('a')
    # raise GoodExc('a')
    return i

pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
    arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
    # set a timeout in order to be able to catch C-c
    r.get(1e100)
except KeyboardInterrupt:
    pass
print results
msg150981 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-01-09 21:27
This is not specific to multiprocessing.  It is really an issue with the pickling of exceptions:

  >>> import cPickle
  >>> class BadExc(Exception):
  ...     def __init__(self, a):
  ...         '''Non-optional param in the constructor.'''
  ...         self.a = a
  ...
  >>> a = cPickle.dumps(BadExc(1))
  >>> cPickle.loads(a)
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.BadExc'>, ())

I think that when you create a new exception class with an __init__() method, you need to make sure that self.args is set properly by calling the __init__() method of the parent class using the same arguments.  So you can instead do

  class BadExc(Exception):
      def __init__(self, a):
          '''Non-optional param in the constructor.'''
          Exception.__init__(self, a)
          self.a = a
msg151003 - (view) Author: Faheem Mitha (fmitha) Date: 2012-01-10 05:22
Thanks to sbt for his helpful and clear explanation.
The following bug report appears relevant, http://bugs.python.org/issue1692335. It seems the proposed fix
was never applied because it caused breakage to existing gcode.
It is not clear to me whether this behavior is considered a bug
or a feature. :-)
msg151228 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2012-01-14 05:23
2.6 and 3.1 only get security fixes.
I am only guessing that this is still an issue for 3.2. A test with 3.2.2 would be good. If this is really about pickling behavior w/r/t exceptions that cannot be changed then this should be closed.
msg151232 - (view) Author: Faheem Mitha (fmitha) Date: 2012-01-14 05:40
This is an issue with SQLAlchemy exceptions, and has been worked around by Mike Bayer in http://www.sqlalchemy.org/trac/ticket/2371

For the record, I think the real problem is that Python exception pickling is broken, see http://bugs.python.org/issue1692335

It would be nice to see this fixed, otherwise this issue will continue to plague the Python standard libraries as well as other libraries. Library writers don't seem to be aware of the issue.
msg161615 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-05-25 20:05
This is a duplicate of #9244 and #9400 which have been fixed by wrapping unpicklable exceptions in picklable exceptions.

The larger issue of many exception classes being unpicklable, is dealt with in #1692335.
History
Date User Action Args
2012-05-25 20:05:52sbtsetstatus: open -> closed
2012-05-25 20:05:17sbtsetresolution: duplicate
messages: + msg161615
stage: resolved
2012-01-14 05:40:43fmithasetmessages: + msg151232
2012-01-14 05:23:47terry.reedysetnosy: + terry.reedy, pitrou, alexandre.vassalotti, jnoller

messages: + msg151228
versions: + Python 2.7, Python 3.2, Python 3.3, - Python 2.6
2012-01-10 05:22:57fmithasetmessages: + msg151003
2012-01-09 21:27:58sbtsetnosy: + sbt
messages: + msg150981
2012-01-09 20:23:57fmithacreate