msg111827 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2010-07-28 16:03 |
The attached test program calls apply_async with a function that will raise CalledProcessError. However, when result.get() is called, it raises a TypeError and the program hangs:
$ ./bug.py
ERROR:root:ops
Traceback (most recent call last):
File "./bug.py", line 19, in run_with
dW1 = run_dcon()
File "./bug.py", line 26, in run_dcon
subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
File "/usr/lib/python2.6/subprocess.py", line 498, in check_call
raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['dcon']' returned non-zero exit status 127
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/usr/lib/python2.6/threading.py", line 484, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib/python2.6/multiprocessing/pool.py", line 259, in _handle_results
task = get()
TypeError: ('__init__() takes exactly 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
|
msg112118 - (view) |
Author: ysj.ray (ysj.ray) |
Date: 2010-07-31 07:48 |
I got "OSError: [Errno 2] No such file or directory" instead of TypeError when running the "bug.py", did I miss something?
|
msg112150 - (view) |
Author: Jesse Noller (jnoller) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2010-07-31 13:08 |
@ray - you probably don't have the "dcon" binary on your path. bug.py calls a subprocess call.
|
msg112158 - (view) |
Author: Nikolaus Rath (nikratio) * |
Date: 2010-07-31 15:36 |
@ray: Try it with the following dummy dcon program:
$ cat dcon
#!/bin/sh
exit 127
(and change the path to dcon in bug.py accordingly).
|
msg112329 - (view) |
Author: ysj.ray (ysj.ray) |
Date: 2010-08-01 14:04 |
This is because when an subprocess.CalledProcessError is raised, the CalledProcessError instance is picked through a socket and then read by "parent" process, but in fact CalledProcessError instances cannot be picked correctly. Because CalledProcessError extends PyExc_BaseException, which defines a __reduce__ method, that special method cause the pickle load to call the exception type's __init__ method with packed self.args as arguments. So if a subclass of "Exception" needs to behave correctly in pickling, it should make self.args meats its __init__ method's function signature. That is, ensure the calling to:
self.__init__(*self.args)
has no problem.
But CalledProcessError doesn't meat this requirement. Here is a patch fixing the CalledProcessError.__init__ to call its base class's __init__ to store its arguments in self.args, which can fix this bug.
|
msg157433 - (view) |
Author: Daniel Blanchard (Daniel.Blanchard) |
Date: 2012-04-03 17:52 |
I believe I'm still encountering this issue in 2.7:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/opt/python/2.7/lib/python2.7/threading.py", line 552, in __bootstrap_inner
self.run()
File "/opt/python/2.7/lib/python2.7/threading.py", line 505, in run
self.__target(*self.__args, **self.__kwargs)
File "/opt/python/2.7/lib/python2.7/multiprocessing/pool.py", line 347, in _handle_results
task = get()
TypeError: ('__init__() takes at least 3 arguments (1 given)', <class 'subprocess.CalledProcessError'>, ())
|
msg157444 - (view) |
Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2012-04-03 21:11 |
Here is another patch, with test.
|
msg159733 - (view) |
Author: Daniel Blanchard (Daniel.Blanchard) |
Date: 2012-04-30 19:59 |
The patch appears to fix the issue, so is there any chance of this actually getting accepted this time? It seems bizarre that such a simple bug that has been on the books for almost two years now can't get a patch accepted.
|
msg159752 - (view) |
Author: Richard Oudkerk (sbt) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2012-05-01 11:57 |
There are plenty of other "bad" exception classes apart from CalledProcessError, including TimeoutExpired in the same file. In fact I suspect this is true of the majority of the exception classes in the stdlib which override __init__. So I am not sure how much good it would do to fix just one example.
Python 3.x's Pool wraps bad exception instances in a MaybeEncodingError class which at least lets you see a stringification of the original exception. I am not sure whether you would want to see a backport of this. Even though 2.7 is in bug fix mode, I think a backport would still be appropriate since it stops a pickling error from killing a worker process, causing a hang.
|
msg159799 - (view) |
Author: Roundup Robot (python-dev) ![Python triager (Python triager)](@@file/triager.png) |
Date: 2012-05-02 15:39 |
New changeset 26bbff4562a7 by Richard Oudkerk in branch '2.7':
Issue #9400: Partial backport of fix for #9244
http://hg.python.org/cpython/rev/26bbff4562a7
|
msg159804 - (view) |
Author: Richard Oudkerk (sbt) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2012-05-02 15:49 |
I have backported the fix for issue #9244 to 2.7. This should fix the hang and produce a traceback containing a representation of the original error.
|
msg176431 - (view) |
Author: Richard Oudkerk (sbt) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2012-11-26 17:30 |
I am reopening this issue because 26bbff4562a7 only dealt with objects which cannot be pickled. But CalledProcessError instances *can* be pickled: the problem is that the resulting data cannot be unpickled.
Note that in Python 3.3 CalledProcessError can be pickled then unpickled because of the fix for #1692335. But there are other types which will still produce "corrupt" pickles.
|
msg274188 - (view) |
Author: Doug Coleman (Doug Coleman) |
Date: 2016-09-01 23:30 |
Six years later and I'm still running into this exact bug with ``subprocess.CalledProcessError`` on python 2.7.12 when doing a ``multiprocessing.Pool.map`` and trying to catch errors from ``subprocess.check_output``.
What's the reason it was never fixed and backported?
|
msg274630 - (view) |
Author: Stanislaw Pitucha (viraptor) |
Date: 2016-09-06 21:07 |
Confirming this on python 2.7.12 on DragonflyBSD
|
msg274636 - (view) |
Author: Davin Potts (davin) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2016-09-06 21:34 |
For clarity:
* Confirmed, can repro on 2.7.12 on OSX.
* Also confirmed that this can not be reproduced on the 3.5 or 3.6 branches.
|
msg274638 - (view) |
Author: Davin Potts (davin) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2016-09-06 21:54 |
Reading through issue1692335 provides a sense of the concerns surrounding the patches they applied to the 3.x branches. Attempting to backport those patches to the 2.7 branch involves non-trivial risk, magnified by the now numerous differences between those branches.
If we had a reasonable workaround to use in 2.7, that would simplify things.
Example workaround that produces behavior in 2.7.12 very like what's observed when using 3.5 or 3.6:
try:
subprocess.check_call(['dcon'], stdout=fh, stderr=fh)
except subprocess.CalledProcessError as cpe:
raise Exception(str(cpe))
|
msg275423 - (view) |
Author: Davin Potts (davin) * ![Python committer (Python committer)](@@file/committer.png) |
Date: 2016-09-09 20:19 |
Richard already backported the fix for objects which can not be pickled but the second fix (applied to 3.3 in #1692335) would be problematic to similarly backport to 2.7. Because there are reasonable workarounds for this in 2.7, going ahead with marking this closed.
|
msg333232 - (view) |
Author: Marc Schlaich (schlamar) * |
Date: 2019-01-08 14:41 |
Davin, why isn't just one of the simple one-line patches applied. This would be much more reasonable instead of closing this as won't fix.
This is a really ugly bug and it took me hours to find the cause...
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:04 | admin | set | github: 53646 |
2019-01-08 14:41:38 | schlamar | set | nosy:
+ schlamar messages:
+ msg333232
|
2016-09-09 20:19:57 | davin | set | status: open -> closed resolution: wont fix messages:
+ msg275423
|
2016-09-06 21:54:35 | davin | set | assignee: sbt -> davin messages:
+ msg274638 |
2016-09-06 21:34:03 | davin | set | messages:
+ msg274636 versions:
- Python 3.5, Python 3.6 |
2016-09-06 21:07:55 | viraptor | set | nosy:
+ viraptor messages:
+ msg274630
|
2016-09-01 23:30:19 | Doug Coleman | set | nosy:
+ Doug Coleman messages:
+ msg274188
|
2016-06-05 20:42:53 | ned.deily | set | nosy:
+ davin
versions:
+ Python 3.5, Python 3.6, - Python 3.2, Python 3.3, Python 3.4 |
2016-04-25 21:19:06 | mdengler | set | nosy:
amaury.forgeotdarc, bronger, jnoller, nikratio, ysj.ray, python-dev, sbt, Daniel.Blanchard, mdengler |
2014-05-12 13:57:03 | mdengler | set | nosy:
+ mdengler
|
2013-02-12 09:57:08 | bronger | set | nosy:
+ bronger
|
2012-11-26 17:37:49 | sbt | link | issue16558 superseder |
2012-11-26 17:30:33 | sbt | set | status: closed -> open versions:
+ Python 3.2, Python 3.3, Python 3.4 messages:
+ msg176431
assignee: sbt resolution: fixed -> (no value) stage: resolved -> needs patch |
2012-05-25 12:15:41 | sbt | set | status: open -> closed resolution: fixed stage: patch review -> resolved |
2012-05-02 15:49:50 | sbt | set | messages:
+ msg159804 |
2012-05-02 15:39:17 | python-dev | set | nosy:
+ python-dev messages:
+ msg159799
|
2012-05-01 11:57:46 | sbt | set | messages:
+ msg159752 |
2012-04-30 23:04:45 | pitrou | set | nosy:
+ sbt
|
2012-04-30 19:59:07 | Daniel.Blanchard | set | messages:
+ msg159733 |
2012-04-03 21:11:08 | amaury.forgeotdarc | set | files:
+ picklable_process_exception.patch
nosy:
+ amaury.forgeotdarc messages:
+ msg157444
stage: test needed -> patch review |
2012-04-03 17:52:25 | Daniel.Blanchard | set | nosy:
+ Daniel.Blanchard
messages:
+ msg157433 versions:
+ Python 2.7, - Python 2.6 |
2010-08-01 14:04:27 | ysj.ray | set | files:
+ issue9400.diff keywords:
+ patch messages:
+ msg112329
|
2010-07-31 15:36:33 | nikratio | set | messages:
+ msg112158 |
2010-07-31 13:08:45 | jnoller | set | messages:
+ msg112150 |
2010-07-31 07:48:15 | ysj.ray | set | nosy:
+ ysj.ray messages:
+ msg112118
|
2010-07-29 02:45:04 | r.david.murray | set | nosy:
+ jnoller
stage: test needed |
2010-07-28 16:03:40 | nikratio | create | |