classification
Title: `concurrent.futures.ProcessPoolExecutor` swallows tracebacks
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Claudiu.Popa, bquinlan, cool-RR, josh.r, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2014-06-20 22:13 by cool-RR, last changed 2015-01-17 19:03 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
issue21817.patch Claudiu.Popa, 2014-06-22 05:47 review
issue21817_1.patch Claudiu.Popa, 2015-01-17 16:15
issue21817_2.patch Claudiu.Popa, 2015-01-17 18:05
Messages (9)
msg221128 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-06-20 22:13
When you use `concurrent.futures.ProcessPoolExecutor` and an exception is raised in the created process, it doesn't show you the traceback. This makes debugging very annoying.

Example file:

    #!python
    
    import sys
    import concurrent.futures 
    
    def f():
        # Successful assert:
        assert True
        return g()
    
    def g():
        # Hard-to-find failing assert:
        assert False
    
        
    if __name__ == '__main__':
        with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
            assert isinstance(executor, concurrent.futures.Executor)
            future = executor.submit(f)
            future.result()
        print('Main process finished')
        
        
If you run this under Windows, you get this: 

    Traceback (most recent call last):
      File "./bug.py", line 20, in <module>
        future.result()
      File "c:\python34\lib\concurrent\futures\_base.py", line 402, in result
        return self.__get_result()
      File "c:\python34\lib\concurrent\futures\_base.py", line 354, in __get_result
        raise self._exception
    AssertionError

This is the traceback of the main process, while we want the traceback of the process that failed.
msg221218 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-06-22 05:47
Hello. Here's a patch based on c4f92b597074, which adds something similar to multiprocessing.pool.
The output after the patch is:

concurrent.futures.process.RemoteTraceback:
"""
Traceback (most recent call last):
  File "D:\Projects\cpython\lib\concurrent\futures\process.py", line 153, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "D:\Projects\cpython\PCbuild\a.py", line 9, in f
    return g()
  File "D:\Projects\cpython\PCbuild\a.py", line 13, in g
    assert False
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "a.py", line 20, in <module>
    future.result()
  File "D:\Projects\cpython\lib\concurrent\futures\_base.py", line 402, in result
    return self.__get_result()
  File "D:\Projects\cpython\lib\concurrent\futures\_base.py", line 354, in __get_result
    raise self._exception
AssertionError


It's a little better than the current output, even though it's a little verbose.
msg225649 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2014-08-22 07:36
Any type of feedback regarding this approach will be appreciated.
msg225660 - (view) Author: Ram Rachum (cool-RR) * Date: 2014-08-22 11:43
Hi Claudiu, sorry for the silence.

This output looks great. I'd love to see that go into Python.
msg234174 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2015-01-17 16:15
Here's the updated version of this patch.
msg234181 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-17 17:54
Hi, here are some comments about the patch:

- RemoteTraceback, ExceptionWithTraceback, rebuild_exc should be private (i.e. with a leading underscore)
- in test_traceback, you can use the context manager form of assertRaises instead of the try..except..else construct
msg234186 - (view) Author: Claudiu Popa (Claudiu.Popa) * Date: 2015-01-17 18:05
Thanks, Antoine! Here's the new version, with your comments addressed.
msg234191 - (view) Author: Roundup Robot (python-dev) Date: 2015-01-17 19:02
New changeset a36b402b099b by Antoine Pitrou in branch 'default':
Issue #21817: When an exception is raised in a task submitted to a ProcessPoolExecutor, the remote traceback is now displayed in the parent process.
https://hg.python.org/cpython/rev/a36b402b099b
msg234192 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-01-17 19:03
Thank you very much. Everything is now committed.
History
Date User Action Args
2015-01-17 19:03:59pitrousetstatus: open -> closed
resolution: fixed
messages: + msg234192

stage: patch review -> resolved
2015-01-17 19:02:48python-devsetnosy: + python-dev
messages: + msg234191
2015-01-17 18:05:08Claudiu.Popasetfiles: + issue21817_2.patch

messages: + msg234186
2015-01-17 17:54:35pitrousetnosy: + pitrou
messages: + msg234181
2015-01-17 16:15:32Claudiu.Popasetfiles: + issue21817_1.patch
type: behavior -> enhancement
messages: + msg234174

versions: - Python 3.4
2014-08-22 11:43:48cool-RRsetmessages: + msg225660
2014-08-22 07:36:34Claudiu.Popasetmessages: + msg225649
2014-06-22 05:47:40Claudiu.Popasetfiles: + issue21817.patch

versions: + Python 3.5
keywords: + patch
nosy: + Claudiu.Popa

messages: + msg221218
stage: patch review
2014-06-21 19:58:25ned.deilysetnosy: + bquinlan
2014-06-21 01:06:15josh.rsetnosy: + josh.r
2014-06-20 22:13:22cool-RRcreate