classification
Title: Printing RecursionError results in RecursionError
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution: duplicate
Dependencies: Superseder: asyncio produces an unexpected traceback with recursive __getattribute__
View: 42848
Assigned To: Nosy List: iritkatriel, terry.reedy, vlad2
Priority: normal Keywords: easy, patch

Created on 2021-01-28 01:45 by vlad2, last changed 2021-02-06 05:32 by vlad2.

Files
File name Uploaded Description Edit
exc.py vlad2, 2021-01-28 01:45 minimal python file reconstruction
Pull Requests
URL Status Linked Edit
PR 24460 open vlad2, 2021-02-06 05:32
Messages (10)
msg385828 - (view) Author: Vladimir Feinberg (vlad2) * Date: 2021-01-28 01:45
Python's standard logger provides an exception() method, which is to be called from except blocks to log out exception traces by pulling from sys.exc_info().

Per https://github.com/python/cpython/blob/3.9/Lib/logging/__init__.py#L617 , logger.exception('my message') eventually boils down to something like traceback.print_exception(ei[0], ei[1], ei[2], None, some_buffer) where ei is the sys.exc_info().

However, the traceback code generates that printout by constructing a TracebackException recursively for every `except` context.

In turn, if a RecursionError was generated from an exception thrown across many except blocks, the TracebackException construction itself will have a RecursionError. This is particularly bad in cases where you'd want to capture this failure information, such as when you're printing the exception, since you'll never find out about the originating error.

Certain (well-used) libraries rely on try/except for control flow, and occasionally do recurse in their except blocks, so such exceptions are not hypothetical.

A solution to this might be to avoid constructing a TracebackException, and instead unwind traceback context using a while loop.
msg385847 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-01-28 06:52
This looks like a duplicate of issue 42848 which was fixed in 3.10.
msg385938 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-29 20:56
With 3.9, exc.py produces for me

Traceback (most recent call last):
  File "F:\Python\a\tem3.py", line 3, in f
    raise ValueError('hello')
ValueError: hello

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\Python\a\tem3.py", line 3, in f
    raise ValueError('hello')
ValueError: hello
...
During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\Python\a\tem3.py", line 11, in <module>
    f()
  File "F:\Python\a\tem3.py", line 5, in f
    f()
  File "F:\Python\a\tem3.py", line 5, in f
    f()
  File "F:\Python\a\tem3.py", line 5, in f
    f()
  [Previous line repeated 992 more times]
  File "F:\Python\a\tem3.py", line 3, in f
    raise ValueError('hello')
RecursionError: maximum recursion depth exceeded while calling a Python object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Programs\Python310\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Programs\Python310\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\Python\a\tem3.py", line 14, in <module>
    print_exception(exc_info[0], exc_info[1], exc_info[2], None)
  File "C:\Programs\Python310\lib\traceback.py", line 113, in print_exception
    for line in TracebackException(
  File "C:\Programs\Python310\lib\traceback.py", line 503, in __init__
    context = TracebackException(
  File "C:\Programs\Python310\lib\traceback.py", line 503, in __init__
    context = TracebackException(
  File "C:\Programs\Python310\lib\traceback.py", line 503, in __init__
    context = TracebackException(
  [Previous line repeated 494 more times]
RecursionError: maximum recursion depth exceeded

This ends the same way as in #42848.


The patch in #42848 will appear in 3.10.0a5 in March.  Guido and Irit decided not to backport at the patch is as much a refactoring as a fix.

On a fresh repository build, exc.py ends with 

Traceback (most recent call last):
  File "f:\python\a\tem3.py", line 3, in f
    raise ValueError('hello')
ValueError: hello

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "f:\python\a\tem3.py", line 11, in <module>
    f()
  File "f:\python\a\tem3.py", line 5, in f
    f()
  File "f:\python\a\tem3.py", line 5, in f
    f()
  File "f:\python\a\tem3.py", line 5, in f
    f()
  [Previous line repeated 995 more times]
  File "f:\python\a\tem3.py", line 3, in f
    raise ValueError('hello')
RecursionError: maximum recursion depth exceeded while calling a Python object
got to the finish line!

I believe this is the expected improvement.
msg385949 - (view) Author: Vladimir Feinberg (vlad2) * Date: 2021-01-29 21:54
I agree with both the duplicate classification and am glad the fix works in 3.10. Thanks all for the responses.

Given the issue can be triggered with a fairly benign setup (pandas triggers such an error, and logger.exception is idiomatic), I do think backport should be worth consideration, but maybe I'm overindexing on the frequency of such exceptions in the wild.
msg385991 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-01-30 17:59
Maybe it is worth doing something less invasive in 3.9, such as catching the recursion error and truncating the output?
msg386082 - (view) Author: Vladimir Feinberg (vlad2) * Date: 2021-02-01 17:26
A simple catch may not work (the very first TracebackException is the one
that gets the RecursionError during initialization of its __context__), but
one thing I was thinking about was walking the __context__ pointers and
nulling out anything past the sys.getrecursionlimit() with a warning.

On Sat, Jan 30, 2021 at 10:00 AM Irit Katriel <report@bugs.python.org>
wrote:

>
> Change by Irit Katriel <iritkatriel@yahoo.com>:
>
>
> ----------
> keywords: +easy
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue43048>
> _______________________________________
>
msg386125 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-02-02 00:38
I meant to catch the exception in the constructor’s recursive call, and if necessary then the same again in format (if there are more function calls per exception in format, it will be necessary. The unit test from the 3.10 patch will tell).

Would that not work?
msg386127 - (view) Author: Vladimir Feinberg (vlad2) * Date: 2021-02-02 01:28
Oh, yes, I suppose, that'll truncate to just the first TracebackException.

On Mon, Feb 1, 2021 at 4:38 PM Irit Katriel <report@bugs.python.org> wrote:

>
> Irit Katriel <iritkatriel@yahoo.com> added the comment:
>
> I meant to catch the exception in the constructor’s recursive call, and if
> necessary then the same again in format (if there are more function calls
> per exception in format, it will be necessary. The unit test from the 3.10
> patch will tell).
>
> Would that not work?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue43048>
> _______________________________________
>
msg386132 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-02-02 08:35
It should truncate at the call that raised the recursion error, not the first one. Do you want to create a patch?
msg386154 - (view) Author: Vladimir Feinberg (vlad2) * Date: 2021-02-02 16:50
Yep, you're right. I'd be happy to (but I've never done it before, so
please give me some time).

On Tue, Feb 2, 2021 at 12:35 AM Irit Katriel <report@bugs.python.org> wrote:

>
> Irit Katriel <iritkatriel@yahoo.com> added the comment:
>
> It should truncate at the call that raised the recursion error, not the
> first one. Do you want to create a patch?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue43048>
> _______________________________________
>
History
Date User Action Args
2021-02-06 05:32:51vlad2setkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request23256
2021-02-02 16:50:55vlad2setmessages: + msg386154
2021-02-02 08:35:40iritkatrielsetmessages: + msg386132
2021-02-02 01:28:38vlad2setmessages: + msg386127
2021-02-02 00:38:04iritkatrielsetmessages: + msg386125
2021-02-01 17:26:07vlad2setmessages: + msg386082
2021-01-30 18:00:11iritkatrielsetkeywords: + easy
2021-01-30 17:59:08iritkatrielsetstatus: closed -> open

messages: + msg385991
versions: - Python 3.10
2021-01-29 21:54:32vlad2setmessages: + msg385949
2021-01-29 20:56:53terry.reedysetstatus: pending -> closed

type: behavior
versions: + Python 3.10, - Python 3.6, Python 3.7
nosy: + terry.reedy

messages: + msg385938
stage: resolved
2021-01-28 12:54:15iritkatrielsetstatus: open -> pending
superseder: asyncio produces an unexpected traceback with recursive __getattribute__
resolution: duplicate
2021-01-28 06:52:44iritkatrielsetnosy: + iritkatriel
messages: + msg385847
2021-01-28 01:45:37vlad2create