classification
Title: threading.Thread should have way to catch an exception thrown within
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Add threading.excepthook() to handle uncaught exceptions raised by Thread.run()
View: 1230540
Assigned To: Nosy List: Joel Croteau, Mark Borgerding, eric.snow, giampaolo.rodola, pablogsal, pitrou, pmpp, rhettinger, tim.peters, vstinner
Priority: normal Keywords:

Created on 2019-04-19 01:36 by Joel Croteau, last changed 2020-09-22 14:48 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
ignored_thread_exc.py Mark Borgerding, 2019-10-29 17:56
Messages (10)
msg340520 - (view) Author: Joel Croteau (Joel Croteau) * Date: 2019-04-19 01:36
This has been commented on numerous times by others (https://stackoverflow.com/questions/2829329/catch-a-threads-exception-in-the-caller-thread-in-python, http://benno.id.au/blog/2012/10/06/python-thread-exceptions, to name a few), but there is no in-built mechanism in threading to catch an unhandled exception thrown by a thread. The default behavior of dumping to stderr is completely useless for error handling in many scenarios. Solutions do exist, but I have yet to see one that is not exceptionally complicated. It seems like checking for exceptions should be a very basic part of any threading library. The simplest solution would be to just have the Thread store any unhandled exceptions and have them raised by Thread.join(). There could also be additional methods to check if exceptions were raised.
msg340542 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2019-04-19 12:04
The current behavior can't be changed for compatibility reasons (imagine user programs starting to raise on Thread.join()), but we could add an option to the threading.Thread() constructor in order to store and propagate exceptions.
msg340556 - (view) Author: Joel Croteau (Joel Croteau) * Date: 2019-04-19 20:41
I agree that we should not change the default behavior of Thread.join(), as that would break existing code, but there are plenty of other ways to do this. I see a couple of possibilities:

1. Add an option to the Thread constructor, something like raise_exc, that defaults to False, but when set to True, causes join() to raise any exceptions.

2. (Better, IMO) Add this option to the join() method instead.

3. Create a new method, join_with_exc(), that acts like join() but raises exceptions from the target.

4. (Should probably do this anyway, regardless of what else we do) Add a new method, check_exc(), that checks if any unhandled exceptions have occurred in the thread and returns and/or raises any that have.
msg340557 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2019-04-19 21:03
Here's a basic decorator along those lines, similar to one that I've used on occasion:

    def as_thread(target):
        def _target():
            try:
                t.result = target()
            except Exception as exc:
                t.failure = exc
        t = threading.Thread(target=_target)
        return t

Sure, it's border-line non-trivial, but I'd hardly call it "exceptionally complicated".

Variations for more flexibility:

    def as_thread(target=None, **tkwds):
        # A decorator to create a one-off thread from a function.
        if target is None:
            # Used as a decorator factory
            return lambda target: as_thread(target, **tkwds)

        def _target(*args, **kwargs):
            try:
                t.result = target(*args, **kwargs)
            except Exception as exc:
                t.failure = exc
        t = threading.Thread(target=_target, **tkwds)
        return t


    def threaded(target, **tkwds):
        # A decorator to produce a started thread when the "function" is called.
        if target is None:
            # Used as a decorator factory
            return lambda target: as_thread(target, **tkwds)

        @functools.wraps(target)
        def wrapper(*targs, **tkwargs)
            def _target(*args, *kwargs):
                try:
                    t.result = target(*args, **kwargs)
                except Exception as exc:
                    t.failure = exc
            t = threading.Thread(target=_target, args=targs, kwargs=tkwargs, **tkwds)
            t.start()
            return t
        return wrapper
msg340558 - (view) Author: Joel Croteau (Joel Croteau) * Date: 2019-04-19 21:07
Yes, I know there are workarounds for it, I have seen many, and everyone seems to have their own version. I'm saying we shouldn't need workarounds though–this should be built in functionality. Ideally, dropping an exception should never be default behavior, but I understand not wanting to break existing code, that's why I'm saying add additional functionality to make these checks easier and not require hacky, un-pythonic wrappers and other methods to find out if your code actually worked.
msg355649 - (view) Author: Mark Borgerding (Mark Borgerding) Date: 2019-10-29 12:29
@pitrou  I don't necessarily agree that "current behavior can't be changed". One major selling point of exceptions is that they cannot be accidentally ignored.  The exception is how the current threading.Thread ignores them.

You are correct that changing Thread.join() so it propagates exceptions by default may break code that relies on the implicit behavior of a thread dying when the target/run method raises.  I'd argue such code deserves to be broken -- "explicit is better than implicit".

I suspect there is more code that will be fixed by such a change than broken.
msg355661 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-10-29 16:27
> I'd argue such code deserves to be broken

That argument falls flat with me and it doesn't show respect for our users.  The proposal would be a major change to the "rules of the game" and would likely require discussion and buy-in on python-dev and perhaps a PEP.

Please heed Antoine's advice.  This is his area of expertise, and he has suggested a plausible way to go forward.
msg355664 - (view) Author: Mark Borgerding (Mark Borgerding) Date: 2019-10-29 17:56
I'm not trying to disrespect anyone: not users nor certainly Python developers.  I have loved Python since I learned it in 2001.  I was merely trying to respond to what seemed like an automatic rejection of changing legacy behavior.  I certainly agree changes to default behavior should not be made lightly, but sometimes they *should* be made.

This may be such a case:
a) A user writing a thread function either consciously thinks about exceptions while they write function or they do not.  Sure, we should always do the former; just like we should always eat our veggies and get lots of exercise, but ....
b) If they *do* think about exceptions, they will either
b.1) write exception handlers within that function, or
b.2) rely on default behavior that simply prints out the exception string to stderr and lets the thread quietly die.
c) If they did not explicitly think about exceptions while they wrote the function, they may have in the back of their mind the reasonable expectation that an exception will not be ignored.

Changing `join()` to propagate unhandled exceptions would admittedly break b.2 code, but provides absolution for all sinners who have written case c.

This is what I meant by, "I suspect there is more code that will be fixed by such a change than broken."

I'll confess I only just now became aware of `threading.excepthook` added in python 3.8.  Does it change this problem?
msg355666 - (view) Author: Joel Croteau (Joel Croteau) * Date: 2019-10-29 18:21
I'm kind of in agreement with Mark on this, actually. I came across this problem when examining some threaded code that was clearly not working as intended, but was reporting success. Figuring out why that was was not easy. The code had been hastily ported to a multi-threaded version from an iterative version by my predecessor, and neither of us had enough familiarity with Python threads to realize what the problem was. The whole point of having exceptions is that it gives you a way of knowing when errors happen without having to add a bunch of extra error checking to your own code. It rather defeats the purpose if code can silently fail while still throwing exceptions, and we have to add extra code to handle special cases like this where they are ignored.
msg377331 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-22 14:48
I consider that this issue as a duplicate of bpo-1230540. Python 3.8 has a new threading.excepthook hook which can be used in various ways to decide how to handle uncatched thread exceptions.

https://docs.python.org/dev/library/threading.html#threading.excepthook
History
Date User Action Args
2020-09-22 14:48:19vstinnersetstatus: open -> closed

superseder: Add threading.excepthook() to handle uncaught exceptions raised by Thread.run()

nosy: + vstinner
messages: + msg377331
resolution: duplicate
stage: resolved
2019-10-29 18:21:51Joel Croteausetmessages: + msg355666
2019-10-29 17:56:00Mark Borgerdingsetfiles: + ignored_thread_exc.py

messages: + msg355664
2019-10-29 16:28:00rhettingersetnosy: + rhettinger
messages: + msg355661
2019-10-29 12:32:07pmppsetnosy: + pmpp
2019-10-29 12:29:57Mark Borgerdingsetnosy: + Mark Borgerding
messages: + msg355649
2019-04-19 21:07:47Joel Croteausetmessages: + msg340558
2019-04-19 21:03:42eric.snowsetnosy: + eric.snow
messages: + msg340557
2019-04-19 20:41:38Joel Croteausetmessages: + msg340556
2019-04-19 12:04:04pitrousetnosy: + giampaolo.rodola, pablogsal
messages: + msg340542
2019-04-19 12:02:34pitrousetnosy: + tim.peters

type: enhancement
versions: + Python 3.8, - Python 3.7
2019-04-19 02:58:26xtreaksetnosy: + pitrou
2019-04-19 01:36:15Joel Croteaucreate