Currently it's quite easy for these errors to go unnoticed.

I'd like a way to easily detect these in CI.

nedbat suggested piping the process output to another tool, and looking for 'Exception ignored in:' but this seems a little diff

Hi Serhiy,

What do you think about this idea?

Normally, we could use -W error when there is an exception.

It looks like a good idea to me. But it should not be -W error. It should be an -X option, and this option should be used exclusively for debugging user extension modules.

> this option should be used exclusively for debugging user extension modules.

I have a very large codebase that fires the odd ResourceWarning, and after fixing them all I'd like to ensure that they do not reoccur. When using `-W error` it still won't fail CI. So I would use this -X option for more than debugging user extension modules

I am working on this issue.

I wrote PR 13187 to control how unraisable exceptions are handled.

Attached uncollectable.py, gc_callback.py and io_destructor.py examples can be
used to test unraisable exceptions.

Without my PR:
---
$ ./python -Werror uncollectable.py
ResourceWarning: gc: 2 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them

$ ./python gc_callback.py
Exception ignored in: <function wr_callback at 0x7f9b48d2b2b0>
Traceback (most recent call last):
  File "gc_callback.py", line 7, in wr_callback
    raise ValueError(42)
ValueError: 42

$ ./python -X dev io_destructor.py
io_destructor.py:4: ResourceWarning: unclosed file <_io.TextIOWrapper name='io_destructor.py' mode='r' encoding='UTF-8'>
  f = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='io_destructor.py' mode='r' encoding='UTF-8'>
OSError: [Errno 9] Bad file descriptor
---

For example, apply attached site_hook.patch to install a custom unraisablehook.

Output with my PR + site_hook.patch:
---
$ ./python -Werror uncollectable.py
ResourceWarning: gc: 2 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them

$ ./python gc_callback.py
Exception ignored in: <function wr_callback at 0x7fb59a0f9510>
  File "gc_callback.py", line 7, in wr_callback
    raise ValueError(42)
ValueError: 42
Traceback (most recent call last):
  File "gc_callback.py", line 11, in <module>
    obj = None

$ ./python -X dev io_destructor.py
io_destructor.py:4: ResourceWarning: unclosed file <_io.TextIOWrapper name='io_destructor.py' mode='r' encoding='UTF-8'>
  f = None
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Exception ignored in: <_io.TextIOWrapper name='io_destructor.py' mode='r' encoding='UTF-8'>
OSError: [Errno 9] Bad file descriptor
Traceback (most recent call last):
  File "io_destructor.py", line 4, in <module>
    f = None
---

The first good news is that it *is* possible to write a custom hook for unraisable for one of the last unraisable exception: _PyGC_DumpShutdownStats() which logs "uncollectable objects at shutdown".

When an unraisable exceptions is logged before Python finalization, the hook can inspect the Python stack to see where the exception has been raised which helps debugging.

too_late_unraisable.py is an example where PyErr_WriteUnraisable() is called very lated during Python finalization to be handled by user code. A destructor (__del__) fails on print() because the stream has been closed.

PyErr_WriteUnraisable() is called by _PyGC_CollectNoFail() at the *end* of PyImport_Cleanup(). At this point, the sys module has already been cleared, as all other modules. A custom sys.unraisablehook cannot be used, because sys has been called.

Ok, let me come back to the initial issue:

Thomas Grainger:
> Currently it's quite easy for these errors to go unnoticed. I'd like a way to easily detect these in CI. nedbat suggested piping the process output to another tool, and looking for 'Exception ignored in:' but this seems a little diff

When PyErr_WriteUnraisable() is called before Python finalization, my PR 13187 allows to handle these exceptions: log them in a dedicated file, abort the process, maybe even open a network connection, etc. The hook allows to implement your chosen behavior.

The problem is more during Python finalization: see attached too_late_unraisable.py example and my previous comment. If PyErr_WriteUnraisable() is called after sys.stderr is closed or closed to None, the function does nothing: the exception is not logged.

The question now becomes: do *all* calls to PyErr_WriteUnraisable() must abort the process? What is the point? Only a very low level debugger like gdb can be used to see the exception.

@Thomas Grainger: Do you want to have to use gdb to trace such very level exception?

IMHO sadly when PyErr_WriteUnraisable() is called way too late, we should simply ignore such exceptions. And so my PR 13187 is good enough to cover most cases.

If someone cares about exceptions raised very late during Python finalization, Python finalization should be enhanced. But this code is very fragile and is not deterministic. It is a work in progress for years to enhance it.

The point for me is that CI will fail if it happens, then I can use gdb to
find out the cause

On Thu, 9 May 2019, 23:17 STINNER Victor, <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> Ok, let me come back to the initial issue:
>
> Thomas Grainger:
> > Currently it's quite easy for these errors to go unnoticed. I'd like a
> way to easily detect these in CI. nedbat suggested piping the process
> output to another tool, and looking for 'Exception ignored in:' but this
> seems a little diff
>
> When PyErr_WriteUnraisable() is called before Python finalization, my PR
> 13187 allows to handle these exceptions: log them in a dedicated file,
> abort the process, maybe even open a network connection, etc. The hook
> allows to implement your chosen behavior.
>
> The problem is more during Python finalization: see attached
> too_late_unraisable.py example and my previous comment. If
> PyErr_WriteUnraisable() is called after sys.stderr is closed or closed to
> None, the function does nothing: the exception is not logged.
>
> The question now becomes: do *all* calls to PyErr_WriteUnraisable() must
> abort the process? What is the point? Only a very low level debugger like
> gdb can be used to see the exception.
>
> @Thomas Grainger: Do you want to have to use gdb to trace such very level
> exception?
>
> IMHO sadly when PyErr_WriteUnraisable() is called way too late, we should
> simply ignore such exceptions. And so my PR 13187 is good enough to cover
> most cases.
>
> If someone cares about exceptions raised very late during Python
> finalization, Python finalization should be enhanced. But this code is very
> fragile and is not deterministic. It is a work in progress for years to
> enhance it.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue36829>
> _______________________________________
>

> The point for me is that CI will fail if it happens, then I can use gdb to find out the cause

I'm not comfortable with forcing users to use a low-level debugger to debug "unraisable exceptions".

I tried PR 13175 on the test suite by forcing the option to 1: always call Py_FatalError(). Many tests break:

14 tests failed:
    test_asyncio test_cmd_line test_coroutines test_cprofile
    test_exceptions test_generators test_import test_io
    test_raise test_repl test_signal test_ssl test_urllib
    test_yield_from

Examples:

test_error_through_destructor (test.test_io.CBufferedReaderTest) ... Fatal Python error: Unraisable exception

FAIL: test_warn_on_full_buffer (test.test_signal.WakeupSocketSignalTests)
FAIL: test_send_error (test.test_signal.WakeupSocketSignalTests)
FAIL: test_wakeup_write_error (test.test_signal.WakeupSignalTests)

test_unraisable (test.test_exceptions.ExceptionTests) ... Fatal Python error: Unraisable exception

test_generators:

Trying:
    del g
Expecting nothing
Fatal Python error: Unraisable exception

etc.

Unraisable exceptions are bad, but it's really hard to fix all of them. They are too many cases where Python is unable to pass exceptions to the parent.

If you want to make the situation better, maybe we should investigate where Python cannot raise exceptions and try to make it possible.

IMHO my PR 13187 adding sys.unraisablehook() is more usable/reasonable option.

--

Thomas Grainger: which code are you running on your CI? Did you try PR 13175 on your CI? Try it with this additional change, to always crash on PyErr_WriteUnraisable(). Does your CI still pass?

diff --git a/Include/cpython/coreconfig.h b/Include/cpython/coreconfig.h
index 375c0b641d..a775584553 100644
--- a/Include/cpython/coreconfig.h
+++ b/Include/cpython/coreconfig.h
@@ -426,7 +426,8 @@ typedef struct {
         .buffered_stdio = -1, \
         ._install_importlib = 1, \
         .check_hash_pycs_mode = NULL, \
-        ._frozen = -1}
+        ._frozen = -1, \
+        .abort_unraisable = 1}
 /* Note: _PyCoreConfig_INIT sets other fields to 0/NULL */
 
 #ifdef __cplusplus

> I'm not comfortable with forcing users to use a low-level debugger to
debug "unraisable exceptions".

Defaulting to noop when the hook fails means I'll never notice the failure
to be able to debug it

On Fri, 10 May 2019, 00:42 STINNER Victor, <report@bugs.python.org> wrote:

>
> STINNER Victor <vstinner@redhat.com> added the comment:
>
> > The point for me is that CI will fail if it happens, then I can use gdb
> to find out the cause
>
> I'm not comfortable with forcing users to use a low-level debugger to
> debug "unraisable exceptions".
>
> I tried PR 13175 on the test suite by forcing the option to 1: always call
> Py_FatalError(). Many tests break:
>
> 14 tests failed:
>     test_asyncio test_cmd_line test_coroutines test_cprofile
>     test_exceptions test_generators test_import test_io
>     test_raise test_repl test_signal test_ssl test_urllib
>     test_yield_from
>
> Examples:
>
> test_error_through_destructor (test.test_io.CBufferedReaderTest) ... Fatal
> Python error: Unraisable exception
>
> FAIL: test_warn_on_full_buffer (test.test_signal.WakeupSocketSignalTests)
> FAIL: test_send_error (test.test_signal.WakeupSocketSignalTests)
> FAIL: test_wakeup_write_error (test.test_signal.WakeupSignalTests)
>
> test_unraisable (test.test_exceptions.ExceptionTests) ... Fatal Python
> error: Unraisable exception
>
> test_generators:
>
> Trying:
>     del g
> Expecting nothing
> Fatal Python error: Unraisable exception
>
> etc.
>
> Unraisable exceptions are bad, but it's really hard to fix all of them.
> They are too many cases where Python is unable to pass exceptions to the
> parent.
>
> If you want to make the situation better, maybe we should investigate
> where Python cannot raise exceptions and try to make it possible.
>
> IMHO my PR 13187 adding sys.unraisablehook() is more usable/reasonable
> option.
>
> --
>
> Thomas Grainger: which code are you running on your CI? Did you try PR
> 13175 on your CI? Try it with this additional change, to always crash on
> PyErr_WriteUnraisable(). Does your CI still pass?
>
> diff --git a/Include/cpython/coreconfig.h b/Include/cpython/coreconfig.h
> index 375c0b641d..a775584553 100644
> --- a/Include/cpython/coreconfig.h
> +++ b/Include/cpython/coreconfig.h
> @@ -426,7 +426,8 @@ typedef struct {
>          .buffered_stdio = -1, \
>          ._install_importlib = 1, \
>          .check_hash_pycs_mode = NULL, \
> -        ._frozen = -1}
> +        ._frozen = -1, \
> +        .abort_unraisable = 1}
>  /* Note: _PyCoreConfig_INIT sets other fields to 0/NULL */
>
>  #ifdef __cplusplus
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue36829>
> _______________________________________
>

> Defaulting to noop when the hook fails means I'll never notice the failure to be able to debug it

I'm not sure that I understood what you mean here. My PR 13187 logs a message into stderr is custom hook fails with a new exception. Well, then you have to fix your hook ;-)

Another example of hook: hook_file.patch logs unraisable exception into ~/unraisable.txt. Patch written for my latest PR 13187 (with the new 'msg' parameter).

Example of output when running the Python test suite (using multiple processes! ./python -m test -j0 -r):
----------
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Exception ignored in: <function TestContext.test_3611.<locals>.C.__del__ at 0x7f0d5c71eb00>
ZeroDivisionError: division by zero
Traceback (most recent call last):
  ...
  File "/home/vstinner/prog/python/master/Lib/test/test_raise.py", line 463, in test_3611
    f()
  File "/home/vstinner/prog/python/master/Lib/test/test_raise.py", line 456, in f
    del x
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Exception ignored in: <generator object f at 0x7f023c985050>
RuntimeError: generator ignored GeneratorExit
Traceback (most recent call last):
  ...
  File "/home/vstinner/prog/python/master/Lib/test/test_generators.py", line 2207, in test_main
    support.run_doctest(test_generators, verbose)
  ...
  File "<doctest test.test_generators.__test__.coroutine[80]>", line 1, in <module>
    del g
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Exception ignored in: <http.client.HTTPResponse object at 0x7f2afefd00e0>
ValueError: I/O operation on closed file.
Traceback (most recent call last):
  ...
  File "/home/vstinner/prog/python/master/Lib/test/test_urllib.py", line 421, in test_invalid_redirect
    urlopen("http://python.org/")
  File "/home/vstinner/prog/python/master/Lib/unittest/case.py", line 237, in __exit__
    traceback.clear_frames(tb)
  File "/home/vstinner/prog/python/master/Lib/traceback.py", line 220, in clear_frames
    tb.tb_frame.clear()
----------

IMHO such hook is more convenient than killing the process with SIGABRT ;-)

I'm interested to modify regrtest (test runner of the Python test suite) to use sys.unraisablehook(). It would be nice to add an option to display again *all* unraisable exceptions in the test summary, at the end.

I did a similar experimentation for any warnings, so my implementation was fragile because regrtest was hard to extend. That's why I introduced a new TestResult type: to be able to add more fields without breaking all the code. Prevously, a test result was a tuple which was manually unpacked. So adding a new field could break code which wasn't updated to handle new fields.

I started a thread on python-dev to discuss the issue:
https://mail.python.org/pipermail/python-dev/2019-May/157436.html

New changeset ef9d9b63129a2f243591db70e9a2dd53fab95d86 by Victor Stinner in branch 'master':
bpo-36829: Add sys.unraisablehook() (GH-13187)
https://github.com/python/cpython/commit/ef9d9b63129a2f243591db70e9a2dd53fab95d86

Follow-up:

* PR 13487 backports enhancement and bugfix to Python 3.7
* PR 13488 adds _PyErr_WriteUnraisableMsg() and adds 'err_msg' field to sys.unraisablehook
* PR 13490 adds test.support.catch_unraisable_exception()

I merged my PR #13187, so I reject PR #13175.

In the python-dev thread, there is no consensus in favor of -X abortunraisable option. The few people who pronounce them on this option were more against it.
https://mail.python.org/pipermail/python-dev/2019-May/157436.html

At least, you can now very easily reimplement it in a few line of pure Python using the new sys.unraisablehook! For example, add this code to Lib/site.py:
---
if 'abortunraisable' in sys._xoptions:
    import signal
    def abort_hook(unraisable,
                   # keep a reference to continue to work
                   # during Python shutdown
                   raise_signal=signal.raise_signal,
                   SIGABRT=signal.SIGABRT):
        raise_signal(SIGABRT)
    sys.unraisablehook = abort_hook
---

Example with attached gc_callback.py:
---
$ ./python -X dev gc_callback.py 
Exception ignored in: <function wr_callback at 0x7fa973faf870>
Traceback (most recent call last):
  File "gc_callback.py", line 7, in wr_callback
    raise ValueError(42)
ValueError: 42

$ ./python -X abortunraisable gc_callback.py 
Aborted (core dumped)

$ ./python -X abortunraisable -X faulthandler gc_callback.py 
Fatal Python error: Aborted

Current thread 0x00007fed6edc7740 (most recent call first):
  File "/home/vstinner/prog/python/master/Lib/site.py", line 649 in abort_hook
  File "gc_callback.py", line 11 in <module>
Aborted (core dumped)
---

New changeset a58db9628d0c96cc5b863137fed4e432238f8027 by Victor Stinner in branch '3.7':
bpo-36829: Enhance PyErr_WriteUnraisable() (GH-13487)
https://github.com/python/cpython/commit/a58db9628d0c96cc5b863137fed4e432238f8027

New changeset e4d300e07c33a9a77549c62d8687d8fe130c53d5 by Victor Stinner in branch 'master':
bpo-36829: Add test.support.catch_unraisable_exception() (GH-13490)
https://github.com/python/cpython/commit/e4d300e07c33a9a77549c62d8687d8fe130c53d5

In PR 13490, Thomas Grainger proposed a cool context manager:

@contextlib.contextmanager
def throw_unraisable_exceptions():
    unraisable = None
    old_hook = sys.unraisablehook

    def hook(exc):
        nonlocal unraisable
        unraisable = exc

    sys.unraisablehook = hook
    try:
        yield
        if unraisable is not None:
            raise unraisable
    finally:
        unraisable = None
        sys.unraisablehook = old_hook

It allows to raise an unraisable exception :-D Example:

try:
    with support.throw_unraisable_exceptions():
        ...
except Exception as e:
    ... # the exception is now here

I don't need such context manager right now, but I like the fact that it becomes possible to write such context manager :-)

I wrote PR 13512 to use support.catch_unraisable_exception() in test_io.test_error_through_destructor(). But this PR is associated to bpo-18748 since the main change is related to the io module, not sys.unraisablehook ;-)

See also bpo-1230540: "sys.excepthook doesn't work in threads".

New changeset df22c03b93ea4620fdf4a0b3cbbbfa7c645af783 by Victor Stinner in branch 'master':
bpo-36829: PyErr_WriteUnraisable() normalizes exception (GH-13507)
https://github.com/python/cpython/commit/df22c03b93ea4620fdf4a0b3cbbbfa7c645af783

Could test.support.catch_unraisable_exception also be documented at https://docs.python.org/3/library/test.html#module-test.support ?

> Could test.support.catch_unraisable_exception also be documented at https://docs.python.org/3/library/test.html#module-test.support ?

I wrote PR 13554 to document it.

New changeset 6dbbe748e101a173b4cff8aada41e9313e287e0f by Victor Stinner in branch 'master':
bpo-36829: Document test.support.catch_unraisable_exception() (GH-13554)
https://github.com/python/cpython/commit/6dbbe748e101a173b4cff8aada41e9313e287e0f

New changeset 71c52e3048dd07567f0c690eab4e5d57be66f534 by Victor Stinner in branch 'master':
bpo-36829: Add _PyErr_WriteUnraisableMsg() (GH-13488)
https://github.com/python/cpython/commit/71c52e3048dd07567f0c690eab4e5d57be66f534

Ok, the initial issue has been fixed by adding a new sys.unraisablehook() function. You can kill the process with SIGABRT using the recipe I proposed there:
https://bugs.python.org/issue36829#msg343201

As a follow-up, I created bpo-37069: "regrtest: log unraisable exceptions and uncaught thread exceptions".

Thanks Thomas Grainger for reviews and for reporting this interesting issue ;-)

New changeset a85a1d337d26a65036e427341d15e3979f7e9ced by Victor Stinner in branch 'master':
bpo-36829: sys.excepthook and sys.unraisablehook flush (GH-13620)
https://github.com/python/cpython/commit/a85a1d337d26a65036e427341d15e3979f7e9ced

New changeset cdce0574d03005e27b843fc110c54c99c7a76412 by Victor Stinner in branch 'master':
bpo-36829: test_threading: Fix a ref cycle (GH-13752)
https://github.com/python/cpython/commit/cdce0574d03005e27b843fc110c54c99c7a76412

See also bpo-37526: Add support.catch_threading_exception().