classification
Title: sys.excepthook doesn't work in threads
Type: behavior Stage: needs patch
Components: Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: CyberJacob, Decorater, Matt Groth, ellisj, eric.araujo, mwh, nikratio, pitrou, tiagoaoa, tim.peters, undercoveridiot
Priority: normal Keywords:

Created on 2005-06-30 19:06 by ellisj, last changed 2017-09-20 17:58 by pitrou.

Messages (14)
msg25694 - (view) Author: Jonathan Ellis (ellisj) Date: 2005-06-30 19:06
simple script to reproduce:

import sys, threading

def log_exception(*args):
    print 'got exception %s' % (args,)
sys.excepthook = log_exception

def foo():
    a = 1 / 0
threading.Thread(target=foo).start()

Note that a normal traceback is printed instead of "got
exception."
msg25695 - (view) Author: Michael Hudson (mwh) (Python committer) Date: 2007-06-06 11:11
I've just run into this, and it's very annoying.  The stupid part is that threads started with the thread module don't have this problem, it's just a problem with threading.Thread()s trying to be too clever.

I would vote for deleting all the code to do with exception printing in threading.py and letting the C machinery take care of it.
msg25696 - (view) Author: Jonathan Ellis (ellisj) Date: 2007-06-15 02:04
Here is a workaround in the meantime:

def install_thread_excepthook():
    """
    Workaround for sys.excepthook thread bug
    (https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
    Call once from __main__ before creating any threads.
    If using psyco, call psycho.cannotcompile(threading.Thread.run)
    since this replaces a new-style class method.
    """
    import sys
    run_old = threading.Thread.run
    def run(*args, **kwargs):
        try:
            run_old(*args, **kwargs)
        except (KeyboardInterrupt, SystemExit):
            raise
        except:
            sys.excepthook(*sys.exc_info())
    threading.Thread.run = run
msg62933 - (view) Author: Tiago Alves (tiagoaoa) Date: 2008-02-24 20:40
I don't see it as a problem or as the threading module trying to be
"clever". It clearly was a design choice to make the module have its own
exception treatment in order to make it clear in which thread the
exception occurred.
IMHO the decision here should be to implement per thread excepthook's.
msg91243 - (view) Author: Ian Beaver (undercoveridiot) Date: 2009-08-03 21:59
I found that the workaround suggested doesn't work when you have a
subclass of threading.Thread and you want to catch everything in the
module that contains the class to a common log.

Say you have a module with a socket server that spawns a thread on
accept and you want to log anything that tracebacks in the module. This
seems to work:

import sys
import logging
from functools import wraps

def myExceptHook(type, value, tb):
    """ Redirect tracebacks to error log """
    import traceback
    rawreport = traceback.format_exception(type, value, tb)
    report = '\n'.join(rawreport)
    log.error(report)

sys.excepthook = myExceptHook
    
def use_my_excepthook(view):
    """ Redirect any unexpected tracebacks """ 
    @wraps(view)
    def run(*args, **kwargs):
        try:
            return view(*args, **kwargs)
        except:
            sys.excepthook(*sys.exc_info())
    return run


Then in your thread subclass:


class MyThread(threading.Thread):
    def __init__(self, socket_conn):
        threading.Thread.__init__(self)
        self.my_conn = socket_conn

    @use_my_excepthook
    def run(self):
        """ Do stuff """
msg91244 - (view) Author: Ian Beaver (undercoveridiot) Date: 2009-08-03 22:24
Instead of using decorators, this is a slightly simpler modification to
the proposed workaround that allows for any subclassed run method to
also be caught.


def installThreadExcepthook():
    """
    Workaround for sys.excepthook thread bug
    From
http://spyced.blogspot.com/2007/06/workaround-for-sysexcepthook-bug.html
   
(https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1230540&group_id=5470).
    Call once from __main__ before creating any threads.
    If using psyco, call psyco.cannotcompile(threading.Thread.run)
    since this replaces a new-style class method.
    """
    init_old = threading.Thread.__init__
    def init(self, *args, **kwargs):
        init_old(self, *args, **kwargs)
        run_old = self.run
        def run_with_except_hook(*args, **kw):
            try:
                run_old(*args, **kw)
            except (KeyboardInterrupt, SystemExit):
                raise
            except:
                sys.excepthook(*sys.exc_info())
        self.run = run_with_except_hook
    threading.Thread.__init__ = init
msg272002 - (view) Author: Decorater (Decorater) * Date: 2016-08-05 01:55
I too agree that I hate the thread exceptions being printed in the console I would suggest python was to error log it all to a file instead (so it does not spam up the console). I get it a lot with websocket / opus errors and it is annoying because it does not cleanup ffmpeg for me.
msg272003 - (view) Author: Decorater (Decorater) * Date: 2016-08-05 02:04
personally these exceptions in console can be annoying to me as I hate seeing them.

Exception in thread Thread-11:
Traceback (most recent call last):
  File "threading.py", line 914, in _bootstrap_inner
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 150, in run
    super().run()
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 108, in run
    self.player(data)
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 669, in play_audio
    sent = self.socket.sendto(packet, (self.endpoint_ip, self.voice_port))
OSError: [WinError 10055] An operation on a socket could not be performed because the system lacked sufficient buffer space or because a queue was full

Exception in thread Thread-21:
Traceback (most recent call last):
  File "threading.py", line 914, in _bootstrap_inner
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 150, in run
    super().run()
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 108, in run
    self.player(data)
  File "E:\Users\Elsword\Desktop\DecoraterBot\Async\test\resources\Dependencies\discord\voice_client.py", line 669, in play_audio
    sent = self.socket.sendto(packet, (self.endpoint_ip, self.voice_port))
OSError: [WinError 10038] An operation was attempted on something that is not a socket
msg272006 - (view) Author: Decorater (Decorater) * Date: 2016-08-05 02:55
Ok, so I just found out you can bypass thread exceptions by wraping the line that actually runs the threads in a try/except block and then using the logging module to log it to a file instead of the console.
msg288513 - (view) Author: CyberJacob (CyberJacob) Date: 2017-02-24 09:45
Does this affect threads started with the multiprocessing library as well?
msg288514 - (view) Author: CyberJacob (CyberJacob) Date: 2017-02-24 10:33
Just confirmed, this does affect multiprocessing. script to reproduce:

import sys, multiprocessing

def log_exception(*args):
    print 'got exception %s' % (args,)

sys.excepthook = log_exception

def foo():
    a = 1 / 0

multiprocessing.Process(target=foo).start()

again, just a normal traceback instead of "got exception"
msg297221 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 20:17
Apparently the `traceback` module has been used from the start (since changset 7f5013a9a9678e86bff00c97f98e7a92c515b94d) to print the exception and not sys.excepthook.  I presume this is an oversight, as I don't see any reason to prefer `traceback` here.  Tim, any insight?
msg302320 - (view) Author: Matt Groth (Matt Groth) Date: 2017-09-16 05:13
Thank you Antoine Pitrou, I was able to disable this behavior by commenting out some lines of code in 'traceback' and replacing them with the appropriate call to 'sys.excepthook'. Note you also have to comment out a few lines in "Modules/_threadmodule.c" to correspond to this change or else that file complains that there wasn't the expected output printed to stderr.
msg302645 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-09-20 17:58
Matt, do you want to post your patch here?
History
Date User Action Args
2017-09-20 17:58:31pitrousetmessages: + msg302645
2017-09-16 05:13:38Matt Grothsetnosy: + Matt Groth
messages: + msg302320
2017-06-28 20:17:02pitrousetversions: + Python 3.6, Python 3.7, - Python 3.2
nosy: + pitrou, tim.peters

messages: + msg297221

type: behavior
stage: test needed -> needs patch
2017-02-24 10:33:39CyberJacobsetmessages: + msg288514
2017-02-24 09:45:23CyberJacobsetnosy: + CyberJacob
messages: + msg288513
2016-08-05 02:55:32Decoratersetmessages: + msg272006
2016-08-05 02:04:58Decoratersetmessages: + msg272003
versions: + Python 3.5
2016-08-05 01:55:13Decoratersetnosy: + Decorater
messages: + msg272002
2011-09-11 18:55:28nikratiosetnosy: + nikratio
2010-08-04 04:44:18terry.reedysetstage: test needed
versions: + Python 2.7, Python 3.2, - Python 2.6, Python 2.5
2010-02-23 21:22:52eric.araujosetnosy: + eric.araujo
2009-08-03 22:24:03undercoveridiotsetmessages: + msg91244
2009-08-03 21:59:32undercoveridiotsetnosy: + undercoveridiot
messages: + msg91243
2008-02-24 20:40:08tiagoaoasetnosy: + tiagoaoa
messages: + msg62933
2007-11-23 09:20:55christian.heimessetversions: + Python 2.6, Python 2.5, - Python 2.4
2005-06-30 19:06:10ellisjcreate