classification
Title: IDLE not displaying RecursionError tracebacks and hangs
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: miss-islington, serhiy.storchaka, taleinat, terry.reedy
Priority: high Keywords: patch

Created on 2016-04-19 22:39 by terry.reedy, last changed 2019-07-09 19:16 by terry.reedy. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13944 merged taleinat, 2019-06-10 19:51
PR 14621 merged miss-islington, 2019-07-06 12:35
PR 14622 merged miss-islington, 2019-07-06 12:35
PR 14657 merged terry.reedy, 2019-07-09 06:08
PR 14677 merged miss-islington, 2019-07-09 18:37
PR 14678 merged miss-islington, 2019-07-09 18:37
Messages (19)
msg263785 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-19 22:39
Test program:

import sys
sys.setrecursionlimit(20)
def f(): return f()
f()

F:\Python\mypy>python tem.py
Traceback (most recent call last):
  File "tem.py", line 4, in <module>
    f()
  File "tem.py", line 3, in f
    def f(): return f()
...
RecursionError: maximum recursion depth exceeded

In 2.7.11, the error is caught and the user process restarted.

======================= RESTART: F:\Python\mypy\tem.py =======================

=============================== RESTART: Shell ===============================
>>> 

In 3.5.1, the user process hangs, ^C does not work, and a restart explicitly requested either with Shell => Restart or running another program.

This behavior is either peculiar to this test case, or a regression, as I remember getting proper RecursionError tracebacks in the past.
msg263795 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 05:18
In 3.6 just setting the recursion level to 20 produces following output on terminal:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 41515)
Unhandled exception in thread started by 

and a hang.

In 2.7 it produces:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 35043)
Request:  <socket._socketobject object at 0xb70e4a84>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 290, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 318, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 331, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 500, in __init__
    SocketServer.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/SocketServer.py", line 652, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/run.py", line 292, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 280, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-2.7-debug/Lib/idlelib/rpc.py", line 300, in _getresponse
    response = self.pollresponse(myseq, wait)
RuntimeError: maximum recursion depth exceeded

*** Unrecoverable, server exiting!
----------------------------------------

and restarts the shell.

Definitely the 20 limit is too low for IDLE.
msg263797 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 05:50
Experimentally found that minimal recursion level for 2.7 is 29, for 3.2-3.3 is 24, and for 3.4-3.6 is 25.

3.2 produces following output and restart the shell:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 37227)
Request:  <socket.socket object, fd=3, family=2, type=1, proto=0>
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 503, in __init__
    socketserver.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-3.2/Lib/socketserver.py", line 648, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/run.py", line 285, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 280, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 300, in _getresponse
    response = self.pollresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 421, in pollresponse
    self.putmessage(message)
  File "/home/serhiy/py/cpython-3.2/Lib/idlelib/rpc.py", line 324, in putmessage
    s = pickle.dumps(message)
RuntimeError: maximum recursion depth exceeded while pickling an object

*** Unrecoverable, server exiting!
----------------------------------------

3.3 hangs without any terminal output.

3.4 produces the most detailed output and hangs:

----------------------------------------
Unhandled server exception!
Thread: SockThread
Client Address:  ('127.0.0.1', 46394)
Request:  <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 48776), raddr=('127.0.0.1', 46394)>
Traceback (most recent call last):
Exception in thread SockThread:
Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 305, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 331, in process_request
    self.finish_request(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 344, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 508, in __init__
    socketserver.BaseRequestHandler.__init__(self, sock, addr, svr)
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 673, in __init__
    self.handle()
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 318, in handle
    rpc.RPCHandler.getresponse(self, myseq=None, wait=0.05)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 288, in getresponse
    response = self._getresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 308, in _getresponse
    response = self.pollresponse(myseq, wait)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 426, in pollresponse
    self.putmessage(message)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 332, in putmessage
    s = dumps(message)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/rpc.py", line 60, in dumps
    p.dump(obj)
RuntimeError: maximum recursion depth exceeded while pickling an object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/serhiy/py/cpython-3.4/Lib/threading.py", line 911, in _bootstrap_inner
    self.run()
  File "/home/serhiy/py/cpython-3.4/Lib/threading.py", line 859, in run
    self._target(*self._args, **self._kwargs)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 162, in manage_socket
    server.handle_request() # A single request only
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 290, in handle_request
    self._handle_request_noblock()
  File "/home/serhiy/py/cpython-3.4/Lib/socketserver.py", line 307, in _handle_request_noblock
    self.handle_error(request, client_address)
  File "/home/serhiy/py/cpython-3.4/Lib/idlelib/run.py", line 288, in handle_error
    traceback.print_exc(file=erf)
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 252, in print_exc
    print_exception(*sys.exc_info(), limit=limit, file=file, chain=chain)
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 169, in print_exception
    for line in _format_exception_iter(etype, value, tb, limit, chain):
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 153, in _format_exception_iter
    yield from _format_list_iter(_extract_tb_iter(tb, limit=limit))
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 18, in _format_list_iter
    for filename, lineno, name, line in extracted_list:
  File "/home/serhiy/py/cpython-3.4/Lib/traceback.py", line 65, in _extract_tb_or_stack_iter
    line = linecache.getline(filename, lineno, f.f_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 15, in getline
    lines = getlines(filename, module_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 42, in getlines
    return updatecache(filename, module_globals)
  File "/home/serhiy/py/cpython-3.4/Lib/linecache.py", line 130, in updatecache
    with tokenize.open(fullname) as fp:
  File "/home/serhiy/py/cpython-3.4/Lib/tokenize.py", line 458, in open
    text = TextIOWrapper(buffer, encoding, line_buffering=True)
RuntimeError: maximum recursion depth exceeded

3.5 and 3.6 behaves as was described in msg263795.
msg263854 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-20 18:17
Thanks for the investigation.  I copied the limit from a python-ideas example and did not realize that IDLE adds so much to the call stack.  I should add a bit on the subject to the IDLE doc in "3.2. IDLE-console differences".  (It can also impact tracebacks -- as reported on a couple of tracker issues.)

"IDLE adds it own calls to the call stack before (and possibly after) running user code.  Tracebacks are filtered to remove these, but the filtering may not be perfect.  Setting a recursion limit that is too low, such as with sys.setrecursionlimit(20), will result in mysterious malfunctions."
msg263856 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-04-20 18:47
We can monkeypatch sys.setrecursionlimit() to set recursion limit to at least say 50.
msg263860 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-04-20 20:20
Interesting idea.  We avoid the problem by adding something like the following to run.py

_setrecursionlimit = sys.setrecursionlimit
def setrecursionlimit(n):
    _setrecursionlimit(max(n, 50))
sys.setrecursionlimit = setrecursionlimit

This works when entered interactively, and I presume it would within run.py.  For _setrecursionlimit to be accessible from user code (to reverse the monkey patching), it would have to be attached to the function as an attribute.

setrecursionlimit.original = _setrecursionlimit

Though user-friendly for most users, this would make IDLE execute code differently from raw Python.  The builtin has a lower limit, based on the current stack depth, but it raises instead of setting a higher limit.  I presume we could use len(inspect.stack()) to get the current 'recursion depth', and add, say, 30 rather than 3. (The current error message could be more helpful.)

>>> sys.setrecursionlimit(3)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RecursionError: cannot set the recursion limit to 3 at the recursion depth 1: the limit is too low
>>> sys.setrecursionlimit(4)
>>> f()
>>>
>>> def f():
...   print('f')
...   f()
...
>>> f()
>>>

The call to f seems to be silently ignored.  This does not seem helpful.

Whatever we do, I would mention it in a revision of the proposed paragraph above.
msg277927 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2016-10-03 03:58
I ran into this again.  Raising priority.
msg345087 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-06-09 19:56
IMO just updating the documentation would not be enough.

I suggest wrapping sys.setrecursionlimit() and *adding* 30 to the given limit, to offset the call stack depth added by IDLE. This makes more sense to me than max(n, 50), which I imagine would cause a lot of confusion in cases where it comes into effect.

If that's not acceptable, then I suggest wrapping it to raise an informative exception if the given limit is too low.
msg345098 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-06-10 01:25
'max(n, 50)' should have been 'min(n, 50)'  Tal, does you comment avove about the former apply when corrected to the latter?

Both get... and setrecursionlimit should be wrapped in a function, such as 'wrap_recursionlimits', called from run.main, each decorated with functools.wraps.  Both docstrings should be augmented with ... .__doc__ += "\n\nWhen running under IDLE, ..."  This is all that is needed for getrecursionlimit.  The question is to do with setrecursionlimit.

Whatever is chosen should be added to the IDLE doc section mentioned above.

Currently, at least on CPython, n must be >= 1 or ValueError is raised.  I don't like changing that.  I don't have a strong feeling about threshholding versus incrementing, but see my question above.
msg345137 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-06-10 17:23
> 'max(n, 50)' should have been 'min(n, 50)'  Tal, does you comment avove about the former apply when corrected to the latter?

Yes. To use your terms, I prefer incrementing over thresholding.

> This is all that is needed for getrecursionlimit.

If we go with incrementing, then getrecursionlimit should likely decrement the returned value by the same amount, so that setrecursionlimit(getrecursionlimit()) doesn't change the recursion limit.

> Currently, at least on CPython, n must be >= 1 or ValueError is raised.  I don't like changing that.

This is another issue that incrementing avoids.

Since it looks like we're in general agreement, Terry, I'll work up a PR.
msg345138 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-06-10 17:48
Setting the recursion limit too low also causes issues in a plain Python interactive session. With current master (to be 3.9) on Win10, I get issues with values up to 6. In IDLE values up to 20 cause an immediate exception similar to the one described by Serhiy.

Anyways, I think we should increment by 25, to always set "safe" values.
msg345141 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-06-10 20:26
See PR GH-13944.
msg345142 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-06-10 20:38
[Initially written before you posted the PR]
Serhiy's result suggests that 25 might not always be enough and he suggested 50.  Plus IDLE sometimes makes more calls internally.  Lets go with at least 30.

I agree that "get...(set...(n)) == n" (when n is positive, a check we should emulate) and "set...(get...())" leaving limit as it was are desirable.  The docs and docstrings just need to say what we do.  Possible additions.
"This IDLE wrapper adds 30 to prevent possible uninterruptible loops."
"This IDLE wrapper subtracts 30 to compensate for the 30 IDLE adds when setting the limit."

The main reason I can think of for users to play with low recursion limits is to explore the implementation.  They simply cannot get stock CPython behavior when running with IDLE.  So I want the IDLE wrappers to not be completely transparent. For that to be obvious in calltips, the first line should say so.

The main reason I want to intervene is to prevent the not-nice behavior of hanging until a manual restart.  I checked, and it is still possible.

In 3.8.0b1, the min n is not 1 but current stack + 5.
>>> sys.setrecursionlimit(11)
Traceback (most recent call last):
  File "<pyshell#9>", line 1, in <module>
    sys.setrecursionlimit(11)
RecursionError: cannot set the recursion limit to 11 at the recursion depth 6: the limit is too low

Our increment will prevent a user from seeing this, but this is necessary to prevent ...

>>> sys.setrecursionlimit(12)
# hangs, once after printing "Exception ignored in thread started by"
# until manual restart.
# 18 hangs after printing 60 lines of a double exception.
# 20 does not hang and calling 'f' from the original message
# gets a truncated traceback ending with
   [Previous line repeated 12 more times]
msg347429 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2019-07-06 12:35
New changeset fcf1d003bf4f0100c9d0921ff3d70e1127ca1b71 by Tal Einat in branch 'master':
bpo-26806: add 30 to the recursion limit in IDLE's shell (GH-13944)
https://github.com/python/cpython/commit/fcf1d003bf4f0100c9d0921ff3d70e1127ca1b71
msg347430 - (view) Author: miss-islington (miss-islington) Date: 2019-07-06 12:54
New changeset d4af55391f56286ab8d478591017174a5a0a5ce2 by Miss Islington (bot) in branch '3.8':
bpo-26806: add 30 to the recursion limit in IDLE's shell (GH-13944)
https://github.com/python/cpython/commit/d4af55391f56286ab8d478591017174a5a0a5ce2
msg347431 - (view) Author: miss-islington (miss-islington) Date: 2019-07-06 12:56
New changeset d666217b26c373784761e3a84f243f02682bccb1 by Miss Islington (bot) in branch '3.7':
bpo-26806: add 30 to the recursion limit in IDLE's shell (GH-13944)
https://github.com/python/cpython/commit/d666217b26c373784761e3a84f243f02682bccb1
msg347573 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-09 18:37
New changeset 6aeb2fe606408aae14c246470794f1303b3be812 by Terry Jan Reedy in branch 'master':
bpo-26806: IDLE should run without docstrings (#14657)
https://github.com/python/cpython/commit/6aeb2fe606408aae14c246470794f1303b3be812
msg347574 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-09 18:59
New changeset b82188d9bad1774624cb1788dbdec2f0d7d65688 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.8':
bpo-26806: IDLE should run without docstrings (GH-14657) (GH-14677)
https://github.com/python/cpython/commit/b82188d9bad1774624cb1788dbdec2f0d7d65688
msg347575 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-07-09 19:16
New changeset f54f062f68ac82fc3e7c92345b4d5fbe2534dc84 by Terry Jan Reedy (Miss Islington (bot)) in branch '3.7':
bpo-26806: IDLE should run without docstrings (GH-14657) (GH-14678)
https://github.com/python/cpython/commit/f54f062f68ac82fc3e7c92345b4d5fbe2534dc84
History
Date User Action Args
2019-07-09 19:16:48terry.reedysetmessages: + msg347575
2019-07-09 18:59:30terry.reedysetmessages: + msg347574
2019-07-09 18:37:47miss-islingtonsetpull_requests: + pull_request14484
2019-07-09 18:37:39miss-islingtonsetpull_requests: + pull_request14483
2019-07-09 18:37:30terry.reedysetmessages: + msg347573
2019-07-09 06:08:42terry.reedysetpull_requests: + pull_request14464
2019-07-06 12:56:54taleinatsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-07-06 12:56:13miss-islingtonsetmessages: + msg347431
2019-07-06 12:54:21miss-islingtonsetnosy: + miss-islington
messages: + msg347430
2019-07-06 12:35:45miss-islingtonsetpull_requests: + pull_request14436
2019-07-06 12:35:38miss-islingtonsetstage: needs patch -> patch review
pull_requests: + pull_request14435
2019-07-06 12:35:29taleinatsetmessages: + msg347429
2019-06-10 20:38:54terry.reedysetmessages: + msg345142
stage: patch review -> needs patch
2019-06-10 20:26:08taleinatsetmessages: + msg345141
2019-06-10 19:51:00taleinatsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13811
2019-06-10 17:48:58taleinatsetmessages: + msg345138
2019-06-10 17:23:42taleinatsetmessages: + msg345137
2019-06-10 01:25:46terry.reedysetmessages: + msg345098
versions: - Python 2.7
2019-06-09 19:56:45taleinatsetnosy: + taleinat
messages: + msg345087
2019-06-09 19:49:26taleinatsetcomponents: - Documentation
versions: + Python 3.8, Python 3.9, - Python 3.5, Python 3.6
2016-10-03 03:58:48terry.reedysetpriority: normal -> high

title: IDLE not displaying RecursionError tracebacks -> IDLE not displaying RecursionError tracebacks and hangs
messages: + msg277927
versions: + Python 3.7
2016-04-20 20:20:09terry.reedysetmessages: + msg263860
2016-04-20 18:47:26serhiy.storchakasetmessages: + msg263856
2016-04-20 18:17:42terry.reedysetmessages: + msg263854
components: + Documentation, IDLE
stage: test needed -> needs patch
2016-04-20 05:50:01serhiy.storchakasetmessages: + msg263797
2016-04-20 05:18:02serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg263795
2016-04-19 22:39:04terry.reedycreate