classification
Title: Infinite recursion while garbage collecting loops indefinitely
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, dizzy, gregory.p.smith, iritkatriel
Priority: normal Keywords:

Created on 2010-12-30 00:51 by dizzy, last changed 2021-04-16 22:44 by iritkatriel. This issue is now closed.

Files
File name Uploaded Description Edit
unnamed gregory.p.smith, 2010-12-30 17:31
unnamed gregory.p.smith, 2011-01-01 02:18
Messages (7)
msg124896 - (view) Author: Mihai Rusu (dizzy) Date: 2010-12-30 00:51
Hi

While working on some Python code I stumbled on a situation where the Python process seems to hang indefinitely. Further debugging points to the following conclusion: if there is a class that somehow manages to run into an infinite recursion (properly detected by Python which raises the corresponding RuntimeError) while Python is doing garbage collection (so the infinite recursion has to be triggered from __del__ somehow) then Python hangs into an infinite loop apparently retrying to call that __del__ indefinitely. The described behavior happens ONLY when an infinite recursion is detected (ie if I raise my own "RuntimeError" from __del__ then it just logs it and exits).

Program showing the problem:

class A(object):
  def __init__(self):
    raise Exception('init error')
    self.m = 'Hello world'

  def __del__(self):
    #raise RuntimeError('my runtime error')
    self.__del__()

def func():
  h = A()

func()

$ python pyloop.py

Traceback (most recent call last):
  File "pyloop.py", line 15, in <module>
    func()
  File "pyloop.py", line 13, in func
    h = A()
  File "pyloop.py", line 5, in __init__
    raise Exception('init error')
Exception: init error
Exception RuntimeError: 'maximum recursion depth exceeded' in <bound method A.__del__ of <__main__.A object at 0x17bb7d0>> ignored
Exception RuntimeError: 'maximum recursion depth exceeded' in <bound method A.__del__ of <__main__.A object at 0x17bb7d0>> ignored
Exception RuntimeError: 'maximum recursion depth exceeded' in <bound method A.__del__ of <__main__.A object at 0x17bb7d0>> ignored
Exception RuntimeError: 'maximum recursion depth exceeded' in <bound method A.__del__ of <__main__.A object at 0x17bb7d0>> ignored
...

If I uncomment the line raising my RuntimeError instance from __del__ then the exception is logged once and the program exits (as expected).

Please help, thanks.
msg124915 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-12-30 12:27
Normally you should never call __del__, OTOH the issue is the same with a class like::

class A:
    def close(self):
        self.close()
    def __del__(self):
        self.close()

The problem is not with _infinite_ recursion, though; a depth of 47 is enough (but 46 won't show the problem)::

class A:
    def __del__(self):
        self.recurse(47)
    def recurse(self, n):
        if n:
            self.recurse(n-1)
        else:
            raise ValueError

Hint: PyTrash_UNWIND_LEVEL is defined to 50; I checked that recompiling Python with a different value also changes the necessary recursion depth.

There is some nasty interaction between the "Trashcan mechanism" and resurrected objects stored in deep structures. A list is enough, no need to involve frames and exceptions::

class C:
    def __del__(self):
        print ".",
        x = self
        for i in range(49):    # PyTrash_UNWIND_LEVEL-1
            x = [x]
l = [C()]
del l
msg124916 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2010-12-30 12:30
Precision: with new-styles classes (or py3k) the limit is PyTrash_UNWIND_LEVEL-2. This does not change anything to the problem.
msg124921 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2010-12-30 17:31
FWIW, the example pasted in the bug was the smallest one he could come up
with.  in reality we were never calling .__del__() explicitly. We ran into
the problem due to a __del__ method triggering a __getattr__ call and the
__getattr__ ending up in infinite recursion because an attribute it accessed
internally wasn't defined.  That particular bug was fixable by fixing the
__getattr__ to not depend on any instance attributes but it is still a
problem in Python for the interpreter to get into an infinite loop calling
the destructor in that case...
msg124984 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-01-01 00:49
2.6 is finished except for possible security patches.
This should be verified in a current release, preferably 3.2
msg124986 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2011-01-01 02:18
it happens on 3.2 (py3k head).
msg386891 - (view) Author: Irit Katriel (iritkatriel) * (Python triager) Date: 2021-02-12 23:57
I think this issue is out of date.

For Mihai's example I get:

>>> class A(object):
...   def __init__(self):
...     raise Exception('init error')
...     self.m = 'Hello world'
...   def __del__(self):
...     #raise RuntimeError('my runtime error')
...     self.__del__()
...
>>> def func():
...   h = A()
...
>>> func()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in func
  File "<stdin>", line 3, in __init__
Exception: init error
>>>



For Amaury's first example I get what I expect:


>>> class A:
...     def close(self):
...         self.close()
...     def __del__(self):
...         self.close()
...
>>> class A:
...     def close(self):
...         self.close()
...     def __del__(self):
...         self.close()
...
>>> def func():
...   h = A()
...
>>> func()
Exception ignored in: <function A.__del__ at 0x000002921A8E1820>
Traceback (most recent call last):
  File "<stdin>", line 5, in __del__
  File "<stdin>", line 3, in close
  File "<stdin>", line 3, in close
  File "<stdin>", line 3, in close
  [Previous line repeated 994 more times]
RecursionError: maximum recursion depth exceeded
>>>


And for Amaury's trashcan example (even when I increase the list length to well over the trashcan threshold): 

>>> class C:
...     def __del__(self):
...         print('.')
...         x = self
...         for i in range(49):    # PyTrash_UNWIND_LEVEL-1
...             x = [x]
...
>>> l = [C()]
>>> del l
.
>>> class C:
...      def __del__(self):
...          print('.')
...          x = self
...          for i in range(5000):
...             x = [x]
...
>>> l = [C()]
>>> del l
.
>>>
History
Date User Action Args
2021-04-16 22:44:16iritkatrielsetstatus: pending -> closed
stage: needs patch -> resolved
2021-02-12 23:57:56iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg386891

resolution: out of date
2013-06-30 05:53:11terry.reedysetnosy: - terry.reedy
2011-01-01 02:18:03gregory.p.smithsetfiles: + unnamed

messages: + msg124986
nosy: terry.reedy, gregory.p.smith, amaury.forgeotdarc, dizzy
2011-01-01 00:49:35terry.reedysetversions: + Python 2.7, - Python 2.6
nosy: + terry.reedy

messages: + msg124984

stage: needs patch
2010-12-30 17:31:41gregory.p.smithsetfiles: + unnamed

messages: + msg124921
2010-12-30 12:30:35amaury.forgeotdarcsetmessages: + msg124916
2010-12-30 12:27:21amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg124915
2010-12-30 00:51:25dizzycreate