classification
Title: _thread.start_new_thread(): call sys.unraisablehook() to handle uncaught exceptions
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lazka, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2019-05-28 10:15 by vstinner, last changed 2019-05-29 01:03 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13617 merged vstinner, 2019-05-28 10:16
Messages (10)
msg343756 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 10:15
Python 3.8 now has 3 hooks for uncaught exceptions:

* sys.excepthook()
* sys.unraisablehook()
* threading.excepthook()

_thread.start_new_thread() calls none of these hooks, but directly logs the exception.

I propose to modify it to reuse sys.unraisablehook(): see attached PR.

--

Using threading.excepthook() would be another option, but _thread.start_new_thread() is usually wrapped in threading.Thread.run() which already uses threading.excepthook(). It might be surprising to see two bugs from the same thread using threading.excepthook().

Moreover, right now, _thread is the "low-level" API to access threads: it doesn't acces threading. I'm not comfortable by adding an inter-dependency between _thread (low-level) and threading (high-level).

If threading.Thread.run() is correctly written, it should not raise an exception. In the worst case, threading.excepthook should handle the exception. _thread.start_new_thread() should never get an exception raised by threading if threading.excepthook does correctly its job.
msg343757 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 10:19
With PR 13617, the function which raised the exception can be retrieved using unraisable.object.

If we used threading.excepthook, I don't see how we could pass this function into threading.ExceptHookArgs: ExceptHookArgs.thread is expected to be a threading.Thread object.
msg343759 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 10:23
This issue is a follow-up of:

* bpo-36829: Add sys.unraisablehook() to custom how "unraisable exceptions" are logged
* bpo-1230540: Add threading.excepthook() to handle uncaught exceptions raised by Thread.run()


Discussions around _thread.start_new_thead() exception:

Antoine Pitrou:
"""
From the doc: Handle uncaught :func:``Thread.run`` exception.

What if some C extension wants to report uncaught exceptions in C-created threads? Does it have to create a dummy Thread object?

For example, how about threads created by _thread.start_new_thread?
"""

https://github.com/python/cpython/pull/13515#issuecomment-495935350


Serhiy Storchaka:
"If we want to support low-level threads created by start_new_thread() we should call [threading.]excepthook() from t_bootstrap instead of Thread._bootstrap_inner. Otherwise implementing excepthook() as a Thread method would be more convenient."

https://bugs.python.org/issue1230540#msg343748
msg343788 - (view) Author: Christoph Reiter (lazka) * Date: 2019-05-28 15:16
> _thread.start_new_thread() calls none of these hooks, but directly logs the exception.

It calls sys.excepthook() currently:

import _thread
import threading
import sys

done = False
def hook(*args):
    global done
    print(threading.current_thread())
    done = True
sys.excepthook = hook

def worker():
    raise Exception
_thread.start_new_thread(worker, tuple())
while not done:
    pass
msg343805 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 16:44
Output:
---
Unhandled exception in thread started by <function worker at 0x7fae27f212d0>
<_DummyThread(Dummy-1, started daemon 140385971111680)>
---

Ah right, sys.excepthook is called! But "Unhandled exception in thread started by <function worker at 0x7fae27f212d0>" line is always written into stderr, it cannot be avoided :-( And sys.excepthook doesn't provide access to the thread function which caused the issue.
msg343806 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 16:49
I updated my PR description:

_thread.start_new_thread() now logs uncaught exception raised by the
function using sys.unraisablehook(), rather than sys.excepthook(), so
the hook gets access to the function which raised the exception.
msg343815 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-28 20:05
> If threading.Thread.run() is correctly written, it should not raise an exception.

Since the error handling for threading.Thread.run() is written on Python there are many opportunities to get an exception in error handling: KeyboardInterrupt, MemoryError and, at the shutdown stage, maybe NameError, AttributeError or TypeError.
msg343823 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-28 22:14
> Since the error handling for threading.Thread.run() is written on Python there are many opportunities to get an exception in error handling: KeyboardInterrupt, MemoryError and, at the shutdown stage, maybe NameError, AttributeError or TypeError.

Sure, the risk is real, but I tried to minimize it.

NameError and AttributeError "should" not happen: I wrote _make_invoke_excepthook() to prevent name errors. Functions used to invoke threading.excepthook are "cached" in a private namespace. Moreover, the default hook is implemented in C also to reduce the risk of raising a new exception.

Anyway, if threading.excepthook() raises a second exception, sys.excepthook() is now called to handle it ;-) That's a Python 3.8 change. In Python 3.7 and older, new exception was handled by start_new_thread().
msg343845 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-29 00:58
New changeset 8b09500345d998f3ff1e363a5210bc87f42ff306 by Victor Stinner in branch 'master':
bpo-37076: _thread.start_new_thread() calls _PyErr_WriteUnraisableMsg() (GH-13617)
https://github.com/python/cpython/commit/8b09500345d998f3ff1e363a5210bc87f42ff306
msg343846 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-05-29 01:03
Thanks Serhiy for the review!
History
Date User Action Args
2019-05-29 01:03:33vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg343846

stage: patch review -> resolved
2019-05-29 00:58:01vstinnersetmessages: + msg343845
2019-05-28 22:14:44vstinnersetmessages: + msg343823
2019-05-28 20:05:12serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg343815
2019-05-28 16:49:32vstinnersetmessages: + msg343806
2019-05-28 16:44:03vstinnersetmessages: + msg343805
2019-05-28 15:16:37lazkasetnosy: + lazka
messages: + msg343788
2019-05-28 10:23:15vstinnersetmessages: + msg343759
2019-05-28 10:19:12vstinnersetmessages: + msg343757
2019-05-28 10:16:10vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request13519
2019-05-28 10:15:10vstinnercreate