classification
Title: In multiprocessing.pool, ExceptionWithTraceback should derive from Exception
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: sbt Nosy List: myint, python-dev, r.david.murray, sbt
Priority: normal Keywords: patch

Created on 2014-03-19 14:53 by myint, last changed 2014-03-23 18:42 by sbt. This issue is now closed.

Files
File name Uploaded Description Edit
exception_with_traceback.diff myint, 2014-03-19 14:53 review
thread_pool_exception_test.py myint, 2014-03-19 15:51
Messages (6)
msg214101 - (view) Author: Steven Myint (myint) * Date: 2014-03-19 14:53
In multiprocessing.pool, ExceptionWithTraceback is not derived from Exception. Thus when it is raised, we get the exception, "TypeError: exceptions must derive from BaseException". Attached is a patch that fixes this.

See below example traceback that prompted me to notice this:

Traceback (most recent call last):
  File "/Users/myint/Library/Python/3.4/bin/yolk", line 9, in <module>
    load_entry_point('yolk3k==0.7.3', 'console_scripts', 'yolk')()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 1123, in main
    my_yolk.run()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 208, in run
    return getattr(self, action)()
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 240, in show_updates
    for (project_name, version, newest) in _updates(pkg_list, self.pypi):
  File "/Users/myint/Library/Python/3.4/lib/python/site-packages/yolk/cli.py", line 1104, in _updates
    names):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 255, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 594, in get
    raise self._value
TypeError: exceptions must derive from BaseException
msg214102 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2014-03-19 14:58
We must have a missing test case, then.  One should be added.
msg214106 - (view) Author: Steven Myint (myint) * Date: 2014-03-19 15:51
I didn't quite understand the multiprocessing test cases, but attached is a standalone test case that reproduces the problem. The problem only shows up when using ThreadPool.

$ python thread_pool_exception_test.py
Traceback (most recent call last):
  File "thread_pool_exception_test.py", line 7, in <module>
    pool.map(exception, [1])
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 255, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 594, in get
    raise self._value
TypeError: exceptions must derive from BaseException
msg214247 - (view) Author: Steven Myint (myint) * Date: 2014-03-20 15:41
This bug was introduced in #13831.
msg214273 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-03-20 20:28
We should only wrap the exception with ExceptionWithTraceback in the process case where it will be pickled and then unpickled.
msg214571 - (view) Author: Roundup Robot (python-dev) Date: 2014-03-23 12:33
New changeset df6a6951b2c9 by Richard Oudkerk in branch '3.4':
Issue #20980: Stop wrapping exception when using ThreadPool.
http://hg.python.org/cpython/rev/df6a6951b2c9
History
Date User Action Args
2014-03-23 18:42:37sbtsetstatus: open -> closed
resolution: fixed
stage: test needed -> resolved
2014-03-23 12:33:17python-devsetnosy: + python-dev
messages: + msg214571
2014-03-20 20:28:23sbtsetassignee: sbt
messages: + msg214273
2014-03-20 15:41:55myintsetmessages: + msg214247
versions: + Python 3.5
2014-03-19 15:51:40myintsetfiles: + thread_pool_exception_test.py

messages: + msg214106
2014-03-19 15:49:48berker.peksagsetnosy: + sbt
2014-03-19 15:17:57benjamin.petersonsetstage: test needed
2014-03-19 14:58:48r.david.murraysetnosy: + r.david.murray
messages: + msg214102
2014-03-19 14:53:54myintcreate