Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

get method of multiprocessing.pool.Async should return full traceback #58039

Closed
fmitha mannequin opened this issue Jan 19, 2012 · 6 comments
Closed

get method of multiprocessing.pool.Async should return full traceback #58039

fmitha mannequin opened this issue Jan 19, 2012 · 6 comments
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@fmitha
Copy link
Mannequin

fmitha mannequin commented Jan 19, 2012

BPO 13831
Files
  • pool-traceback.patch
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2013-05-06.11:40:27.805>
    created_at = <Date 2012-01-19.20:46:39.394>
    labels = ['type-feature', 'library']
    title = 'get method of  multiprocessing.pool.Async should return full traceback'
    updated_at = <Date 2013-05-06.18:16:59.891>
    user = 'https://bugs.python.org/fmitha'

    bugs.python.org fields:

    activity = <Date 2013-05-06.18:16:59.891>
    actor = 'sbt'
    assignee = 'none'
    closed = True
    closed_date = <Date 2013-05-06.11:40:27.805>
    closer = 'sbt'
    components = ['Library (Lib)']
    creation = <Date 2012-01-19.20:46:39.394>
    creator = 'fmitha'
    dependencies = []
    files = ['30109']
    hgrepos = []
    issue_num = 13831
    keywords = ['patch']
    message_count = 6.0
    messages = ['151651', '187247', '187818', '188276', '188513', '188567']
    nosy_count = 4.0
    nosy_names = ['fmitha', 'slinkp', 'python-dev', 'sbt']
    pr_nums = []
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue13831'
    versions = ['Python 3.4']

    @fmitha
    Copy link
    Mannequin Author

    fmitha mannequin commented Jan 19, 2012

    The documentation in
    http://docs.python.org/library/multiprocessing.html#module-multiprocessing.pool

    says

    """class multiprocessing.pool.AsyncResult¶
    The class of the result returned by Pool.apply_async() and Pool.map_async().

    get([timeout])
    Return the result when it arrives. If timeout is not None and the result does not arrive within timeout seconds then multiprocessing.TimeoutError is raised. If the remote call raised an exception then that exception will be reraised by get()."""

    Consider the example code

    ################################

    from multiprocessing import Pool
    
    def go():
        print 1
        raise Exception("foobar")
        print 2
    
    p = Pool()
    x = p.apply_async(go)
    x.get()
    p.close()
    p.join()

    ###########################

    The traceback from this is

    Traceback (most recent call last):
      File "<stdin>", line 10, in <module>
      File "/usr/lib/python2.6/multiprocessing/pool.py", line 422, in get
        raise self._value
    Exception: foobar
    1

    As is clear in this example, this is not a full traceback - it only shows the traceback to the line where get is located and gives no further information. This is the case in all the other places I have used get. It seems to me that it *should* return the full traceback, which may contain important information missing in such a partial one. I don't know whether one would call this a feature request or a bug report. Maybe there is some technical reason why this is not possible, but I can't think of one.

    @fmitha fmitha mannequin added stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Jan 19, 2012
    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Apr 18, 2013

    Pickling an exception (assuming it works) does not capture the traceback. Doing so would be difficult/impossible since the traceback refers to a linked list of frames, and each frame has references to lots of other stuff like the code object, the global dict, local dict, builtin dict, ... I certainly do not know how to make traceback objects pickle compatible.

    But you could wrap any exception raised by your function in another exception whose representation contains a formatted traceback of the original exception. E.g.

    class WrapException(Exception):
        def __init__(self):
            exc_type, exc_value, exc_tb = sys.exc_info()
            self.exception = exc_value
            self.formatted = ''.join(traceback.format_exception(exc_type, exc_value, exc_tb))
        def __str__(self):
            return '%s\nOriginal traceback:\n%s' % (Exception.__str__(self), self.formatted)
    
    def go():
        try:
            1/0
        except Exception:
            raise WrapException()

    Then raising an unpickled WrapException instance gives the original traceback

    >>> try: go()
    ... except Exception as e: exc = e
    ...
    >>> raise pickle.loads(pickle.dumps(exc))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    __main__.WrapException:
    Original traceback:
    Traceback (most recent call last):
      File "<stdin>", line 3, in go
    ZeroDivisionError: integer division or modulo by zero

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented Apr 25, 2013

    It might be possible to come up with a hack so that when the exception is unpickled in the main process it gets a secondary exception chained to it using __cause__ or __context__ whose stringification contains the stringification of the original traceback.

    @sbt
    Copy link
    Mannequin

    sbt mannequin commented May 2, 2013

    Attached is a patch for 3.4 which uses the __cause__ hack to embed the remote traceback in the local traceback. It will not work for 2.x though.

    >>> import multiprocessing, subprocess
    >>> with multiprocessing.Pool() as p: p.apply(subprocess.Popen, (1,))
    ...
    multiprocessing.pool.RemoteTraceback:
    """
    Traceback (most recent call last):
      File "/home/oudkerk/Repos/py-default/Lib/multiprocessing/pool.py", line 114, in worker
        result = (True, func(*args, **kwds))
      File "/home/oudkerk/Repos/py-default/Lib/subprocess.py", line 838, in __init__
        restore_signals, start_new_session)
      File "/home/oudkerk/Repos/py-default/Lib/subprocess.py", line 1317, in _execute_child
        args = list(args)
    TypeError: 'int' object is not iterable
    """
    
    The above exception was the direct cause of the following exception:
    
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/oudkerk/Repos/py-default/Lib/multiprocessing/pool.py", line 245, in apply
        return self.apply_async(func, args, kwds).get()
      File "/home/oudkerk/Repos/py-default/Lib/multiprocessing/pool.py", line 588, in get
        raise self._value
    TypeError: 'int' object is not iterable

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented May 6, 2013

    New changeset a2928dd2fde4 by Richard Oudkerk in branch 'default':
    Correct issue number for c4f92b597074 in Misc/NEWS from bpo-13813 to bpo-13831
    http://hg.python.org/cpython/rev/a2928dd2fde4

    @sbt sbt mannequin closed this as completed May 6, 2013
    @sbt
    Copy link
    Mannequin

    sbt mannequin commented May 6, 2013

    The relevant changeset was c4f92b597074, but I wrote the wrong issue number in the commit message and Misc/NEWS.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    0 participants