classification
Title: IDLE not displaying RecursionError tracebacks and hangs
Type: behavior Stage: needs patch
Components: Documentation, IDLE Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: serhiy.storchaka, terry.reedy
Priority: high Keywords:

Created on 2016-04-19 22:39 by terry.reedy, last changed 2016-10-03 03:58 by terry.reedy.

Messages (7)
msg263785 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-19 22:39
Test program:

import sys
sys.setrecursionlimit(20)
def f(): return f()
f()

F:\Python\mypy>python tem.py
Traceback (most recent call last):
  File "tem.py", line 4, in <module>
    f()
  File "tem.py", line 3, in f
    def f(): return f()
...
RecursionError: maximum recursion depth exceeded

In 2.7.11, the error is caught and the user process restarted.

======================= RESTART: F:\Python\mypy\tem.py =======================

=============================== RESTART: Shell ===============================
>>> 

In 3.5.1, the user process hangs, ^C does not work, and a restart explicitly requested either with Shell => Restart or running another program.

This behavior is either peculiar to this test case, or a regression, as I remember getting proper RecursionError tracebacks in the past.
msg263795 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 05:18
In 3.6 just setting the recursion level to 20 produces following output on terminal:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 41515)
Unhandled exception in thread started by 

and a hang.

In 2.7 it produces:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 35043)
Request:  <socket._socketobject object at 0xb70e4a84>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 500, in __init__
    SocketServer.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 652, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/run.py", line 292, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 280, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 300, in _getresponse
    response = self.pollresponse(myseq, wait)
RuntimeError: maximum recursion depth exceeded

*** Unrecoverable, server exiting!
----------------------------------------

and restarts the shell.

Definitely the 20 limit is too low for IDLE.
msg263797 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 05:50
Experimentally found that minimal recursion level for 2.7 is 29, for 3.2-3.3 is 24, and for 3.4-3.6 is 25.

3.2 produces following output and restart the shell:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 37227)
Request:  <socket.socket object, fd=3, family=2, type=1, proto=0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 503, in __init__
    socketserver.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 648, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/run.py", line 285, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 280, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 300, in _getresponse
    response = self.pollresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 421, in pollresponse
    self.putmessage(message)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 324, in putmessage
    s = pickle.dumps(message)
RuntimeError: maximum recursion depth exceeded while pickling an object

*** Unrecoverable, server exiting!
----------------------------------------

3.3 hangs without any terminal output.

3.4 produces the most detailed output and hangs:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 46394)
Request:  <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 48776), raddr=('127.0.0.1', 46394)>
Traceback (most recent call last):
Exception in thread SockThread:
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 305, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 331, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 508, in __init__
    socketserver.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 673, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 318, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 288, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 308, in _getresponse
    response = self.pollresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 426, in pollresponse
    self.putmessage(message)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 332, in putmessage
    s = dumps(message)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 60, in dumps
    p.dump(obj)
RuntimeError: maximum recursion depth exceeded while pickling an object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.4/Lib/threading.py", line 911, in _bootstrap_inner
    self.run()
  File "/home/serhiy/py/cpython-3.4/Lib/threading.py", line 859, in run
    self._target(*self._args, **self._kwargs)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 162, in manage_socket
    server.handle_request() # A single request only
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 290, in handle_request
    self._handle_request_noblock()
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 307, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 288, in handle_error
    traceback.print_exc(file=erf)
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 252, in print_exc
    print_exception(*sys.exc_info(), limit=limit, file=file, chain=chain)
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 169, in print_exception
    for line in _format_exception_iter(etype, value, tb, limit, chain):
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 153, in _format_exception_iter
    yield from _format_list_iter(_extract_tb_iter(tb, limit=limit))
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 18, in _format_list_iter
    for filename, lineno, name, line in extracted_list:
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 65, in _extract_tb_or_stack_iter
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 15, in getline
    lines = getlines(filename, module_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 42, in getlines
    return updatecache(filename, module_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 130, in updatecache
    with tokenize.open(fullname) as fp:
  File "/home/serhiy/py/cpython-3.4/Lib/tokenize.py", line 458, in open
    text = TextIOWrapper(buffer, encoding, line_buffering=True)
RuntimeError: maximum recursion depth exceeded

3.5 and 3.6 behaves as was described in msg263795.
msg263854 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-20 18:17
Thanks for the investigation.  I copied the limit from a python-ideas example and did not realize that IDLE adds so much to the call stack.  I should add a bit on the subject to the IDLE doc in "3.2. IDLE-console differences".  (It can also impact tracebacks -- as reported on a couple of tracker issues.)

"IDLE adds it own calls to the call stack before (and possibly after) running user code.  Tracebacks are filtered to remove these, but the filtering may not be perfect.  Setting a recursion limit that is too low, such as with sys.setrecursionlimit(20), will result in mysterious malfunctions."
msg263856 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 18:47
We can monkeypatch sys.setrecursionlimit() to set recursion limit to at least say 50.
msg263860 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-20 20:20
Interesting idea.  We avoid the problem by adding something like the following to run.py

_setrecursionlimit = sys.setrecursionlimit
def setrecursionlimit(n):
    _setrecursionlimit(max(n, 50))
sys.setrecursionlimit = setrecursionlimit

This works when entered interactively, and I presume it would within run.py.  For _setrecursionlimit to be accessible from user code (to reverse the monkey patching), it would have to be attached to the function as an attribute.

setrecursionlimit.original = _setrecursionlimit

Though user-friendly for most users, this would make IDLE execute code differently from raw Python.  The builtin has a lower limit, based on the current stack depth, but it raises instead of setting a higher limit.  I presume we could use len(inspect.stack()) to get the current 'recursion depth', and add, say, 30 rather than 3. (The current error message could be more helpful.)

>>> sys.setrecursionlimit(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RecursionError: cannot set the recursion limit to 3 at the recursion depth 1: the limit is too low
>>> sys.setrecursionlimit(4)
>>> f()
>>>
>>> def f():
...   print('f')
...   f()
...
>>> f()
>>>

The call to f seems to be silently ignored.  This does not seem helpful.

Whatever we do, I would mention it in a revision of the proposed paragraph above.
msg277927 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-10-03 03:58
I ran into this again.  Raising priority.
History
Date User Action Args
2016-10-03 03:58:48terry.reedysetpriority: normal -> high

title: IDLE not displaying RecursionError tracebacks -> IDLE not displaying RecursionError tracebacks and hangs
messages: + msg277927
versions: + Python 3.7
2016-04-20 20:20:09terry.reedysetmessages: + msg263860
2016-04-20 18:47:26serhiy.storchakasetmessages: + msg263856
2016-04-20 18:17:42terry.reedysetmessages: + msg263854
components: + Documentation, IDLE
stage: test needed -> needs patch
2016-04-20 05:50:01serhiy.storchakasetmessages: + msg263797
2016-04-20 05:18:02serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg263795
2016-04-19 22:39:04terry.reedycreate