classification
Title: multiprocessing.pool.AsyncResult.get() messes up exceptions
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: davin Nosy List: Daniel.Blanchard, Doug Coleman, amaury.forgeotdarc, bronger, davin, jnoller, mdengler, nikratio, python-dev, sbt, viraptor, ysj.ray
Priority: normal Keywords: patch

Created on 2010-07-28 16:03 by nikratio, last changed 2016-09-09 20:19 by davin. This issue is now closed.

Files
File name Uploaded Description Edit
bug.py nikratio, 2010-07-28 16:03 Test Case
issue9400.diff ysj.ray, 2010-08-01 14:04 patch against trunk review
picklable_process_exception.patch amaury.forgeotdarc, 2012-04-03 21:11
Messages (17)
msg111827 - (view) Author: Nikolaus Rath (nikratio) * Date: 2010-07-28 16:03
The attached test program calls apply_async with a function that will raise CalledProcessError. However, when result.get() is called, it raises a TypeError and the program hangs:

$ ./bug.py
ERROR:root:ops
Traceback (most recent call last):
  File "./bug.py", line 19, in run_with
    dW1 = run_dcon()
  File "./bug.py", line 26, in run_dcon
    subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
  File "/usr/lib/python2.6/subprocess.py", line 498, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['dcon']' returned non-zero exit status 127
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
    task = get()
TypeError: ('__init__() takes exactly 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
msg112118 - (view) Author: ysj.ray (ysj.ray) Date: 2010-07-31 07:48
I got "OSError: [Errno 2] No such file or directory" instead of TypeError when running the "bug.py", did I miss something?
msg112150 - (view) Author: Jesse Noller (jnoller) * (Python committer) Date: 2010-07-31 13:08
@ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call.
msg112158 - (view) Author: Nikolaus Rath (nikratio) * Date: 2010-07-31 15:36
@ray: Try it with the following dummy dcon program:

$ cat dcon 
#!/bin/sh

exit 127

(and change the path to dcon in bug.py accordingly).
msg112329 - (view) Author: ysj.ray (ysj.ray) Date: 2010-08-01 14:04
This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to:
          self.__init__(*self.args)
has no problem. 

But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug.
msg157433 - (view) Author: Daniel Blanchard (Daniel.Blanchard) Date: 2012-04-03 17:52
I believe I'm still encountering this issue in 2.7:


Exception in thread Thread-3:
Traceback (most recent call last):
  File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
    self.run()
  File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
    task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
msg157444 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2012-04-03 21:11
Here is another patch, with test.
msg159733 - (view) Author: Daniel Blanchard (Daniel.Blanchard) Date: 2012-04-30 19:59
The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted.
msg159752 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-05-01 11:57
There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file.  In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__.  So I am not sure how much good it would do to fix just one example.

Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception.  I am not sure whether you would want to see a backport of this.  Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang.
msg159799 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2012-05-02 15:39
New changeset 26bbff4562a7 by Richard Oudkerk in branch '2.7':
Issue #9400: Partial backport of fix for #9244
http://hg.python.org/cpython/rev/26bbff4562a7
msg159804 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-05-02 15:49
I have backported the fix for issue #9244 to 2.7.  This should fix the hang and produce a traceback containing a representation of the original error.
msg176431 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2012-11-26 17:30
I am reopening this issue because 26bbff4562a7 only dealt with objects which cannot be pickled.  But CalledProcessError instances *can* be pickled: the problem is that the resulting data cannot be unpickled.

Note that in Python 3.3 CalledProcessError can be pickled then unpickled because of the fix for #1692335.  But there are other types which will still produce "corrupt" pickles.
msg274188 - (view) Author: Doug Coleman (Doug Coleman) Date: 2016-09-01 23:30
Six years later and I'm still running into this exact bug with ``subprocess.CalledProcessError`` on python 2.7.12 when doing a ``multiprocessing.Pool.map`` and trying to catch errors from ``subprocess.check_output``. 

What's the reason it was never fixed and backported?
msg274630 - (view) Author: Stanislaw Pitucha (viraptor) Date: 2016-09-06 21:07
Confirming this on python 2.7.12 on DragonflyBSD
msg274636 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-06 21:34
For clarity:
* Confirmed, can repro on 2.7.12 on OSX.
* Also confirmed that this can not be reproduced on the 3.5 or 3.6 branches.
msg274638 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-06 21:54
Reading through issue1692335 provides a sense of the concerns surrounding the patches they applied to the 3.x branches.  Attempting to backport those patches to the 2.7 branch involves non-trivial risk, magnified by the now numerous differences between those branches.

If we had a reasonable workaround to use in 2.7, that would simplify things.

Example workaround that produces behavior in 2.7.12 very like what's observed when using 3.5 or 3.6:
        try:
            subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
        except subprocess.CalledProcessError as cpe:
            raise Exception(str(cpe))
msg275423 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2016-09-09 20:19
Richard already backported the fix for objects which can not be pickled but the second fix (applied to 3.3 in #1692335) would be problematic to similarly backport to 2.7.  Because there are reasonable workarounds for this in 2.7, going ahead with marking this closed.
History
Date User Action Args
2016-09-09 20:19:57davinsetstatus: open -> closed
resolution: wont fix
messages: + msg275423
2016-09-06 21:54:35davinsetassignee: sbt -> davin
messages: + msg274638
2016-09-06 21:34:03davinsetmessages: + msg274636
versions: - Python 3.5, Python 3.6
2016-09-06 21:07:55viraptorsetnosy: + viraptor
messages: + msg274630
2016-09-01 23:30:19Doug Colemansetnosy: + Doug Coleman
messages: + msg274188
2016-06-05 20:42:53ned.deilysetnosy: + davin

versions: + Python 3.5, Python 3.6, - Python 3.2, Python 3.3, Python 3.4
2016-04-25 21:19:06mdenglersetnosy: amaury.forgeotdarc, bronger, jnoller, nikratio, ysj.ray, python-dev, sbt, Daniel.Blanchard, mdengler
2014-05-12 13:57:03mdenglersetnosy: + mdengler
2013-02-12 09:57:08brongersetnosy: + bronger
2012-11-26 17:37:49sbtlinkissue16558 superseder
2012-11-26 17:30:33sbtsetstatus: closed -> open
versions: + Python 3.2, Python 3.3, Python 3.4
messages: + msg176431

assignee: sbt
resolution: fixed -> (no value)
stage: resolved -> needs patch
2012-05-25 12:15:41sbtsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2012-05-02 15:49:50sbtsetmessages: + msg159804
2012-05-02 15:39:17python-devsetnosy: + python-dev
messages: + msg159799
2012-05-01 11:57:46sbtsetmessages: + msg159752
2012-04-30 23:04:45pitrousetnosy: + sbt
2012-04-30 19:59:07Daniel.Blanchardsetmessages: + msg159733
2012-04-03 21:11:08amaury.forgeotdarcsetfiles: + picklable_process_exception.patch

nosy: + amaury.forgeotdarc
messages: + msg157444

stage: test needed -> patch review
2012-04-03 17:52:25Daniel.Blanchardsetnosy: + Daniel.Blanchard

messages: + msg157433
versions: + Python 2.7, - Python 2.6
2010-08-01 14:04:27ysj.raysetfiles: + issue9400.diff
keywords: + patch
messages: + msg112329
2010-07-31 15:36:33nikratiosetmessages: + msg112158
2010-07-31 13:08:45jnollersetmessages: + msg112150
2010-07-31 07:48:15ysj.raysetnosy: + ysj.ray
messages: + msg112118
2010-07-29 02:45:04r.david.murraysetnosy: + jnoller

stage: test needed
2010-07-28 16:03:40nikratiocreate