classification
Title: IDLE ignores sys.excepthook in normal, subprocess mode
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: aroberge, kenny2python, miss-islington, terry.reedy
Priority: normal Keywords: patch

Created on 2021-01-23 11:30 by kenny2python, last changed 2021-02-04 17:48 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
bpo-43008.patch kenny2python, 2021-01-23 11:30 Fix for this issue, patch file from GitHub PR
bpo-43008.patch.2 kenny2python, 2021-01-23 16:19 New patch with requested changes
Pull Requests
URL Status Linked Edit
PR 24302 merged kenny2python, 2021-01-23 11:30
PR 24346 merged miss-islington, 2021-01-26 23:56
PR 24347 merged miss-islington, 2021-01-26 23:56
PR 24370 merged terry.reedy, 2021-01-29 17:32
PR 24374 merged miss-islington, 2021-01-29 18:02
PR 24375 merged miss-islington, 2021-01-29 18:02
Messages (20)
msg385529 - (view) Author: Ken Hilton (kenny2python) * Date: 2021-01-23 11:30
Similar to https://bugs.python.org/issue12643 - IDLE ignores sys.excepthook and always prints exceptions.
msg385540 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 14:31
I looked at all 4 hooks.

sys.breakpointhook is invoked by breakpoint() in user code, so is not IDLE's concern at present.

sys.unraiseablehook is invoked by the interpreter when user code messes up badly, so is not IDLE's condern.  (And it is an expert feature that beginners should ignore.)

sys.displayhook is set to rpc.displayhook in run.py or, if started with -n no subprocess, pyshell.py.  IDLE executes user code with exec().  Shell entries are compiled in 'single' mode.  Exec invokes displayhook for expression statements so compiled.  In both normal and -n mode, displayhook exceptions are displayed  and processed as user code exceptions, with the last traceback line giving the bad line.

sys.excepthook is invoked by the interpreter for uncaught exceptions.
In IDLE's -n mode, sys.excepthook *is* invoked.  If it raises, both tracebacks are printed, separated by "\nDuring handling of the above exception, another exception occurred:\n".  This issue is about having the same effect in normal 2-process mode.

Since, in the subprocess, IDLE catches all user exceptions for its custom handling, there are no uncaught user exceptions.  I believe your patch calls the user hook in the proper place, but an uncaught exception in sys.excepthook would be nasty.  Replace the call with something like

        try:
            sys.e...k(...)
        except BaseException as e:
            print_exception()
            print("\nDuring handling of the above exception, another exception occurred:\n", file=sys.stderr)
            self.usr_exc_info = sys.exc_info()
            print_exception()

If possible, I want a unittest added in test_run.py.  (I am not sure exactly how as I don't think we have yet directly faked receipt of compiled user code.)

class Executive(unittest.TextCase):

    def test_good_excepthook(self):

    def test_bad_excepthook(self):


A blurb is needed (and always is for anything non-trivial).

In Doc/library/idle.rst, I want to insert " and reset display and exception handling" before the period ending the current Shell entry.

Restart Shell
  Restart the shell to clean the environment.
msg385542 - (view) Author: Ken Hilton (kenny2python) * Date: 2021-01-23 16:19
Made requested changes, besides the tests. I didn't manage to figure out how to go about testing an excepthook in the first place, let alone a good or bad one. Hope that's not an issue.
msg385547 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 17:04
Andre: while I would like friendlier tracebacks available in IDLE without 3rd party installation, I doubt this will happen soon.  In the meanwhile, I see that you are working to polish Friendly Traceback with a friendly contest.  I presume FT uses sys.excepthook.  If so, this issue is opportune for supporting your package.  Feel free to test the minimal patch as is, or when fleshed out, and let us know how it works.
msg385548 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 17:09
Ken: I will probably work on tests, perhaps as a followup or parallel issue.  I will likely do a bit or refactoring to make testing easier.

I just looked at print_exception and discovered that the double traceback can likely by handled by properly setting exc.context for the 2nd exception.  But getting that right will be aided by having something that already works.

Possible manual test protocol in Shell

>>> import sys
>>> def egood(a,b,c): print('Exception traceback', file=sys.stderr)

>>> sys.excepthook = egood
>>> 1/0
# "Exception traceback"
>>> def ebad(a,b,c): z

>>> sys.excepthook = ebad
>>> 1/0
# Double traceback for ZeroDivisionError and NameError
msg385550 - (view) Author: Andre Roberge (aroberge) * Date: 2021-01-23 17:14
Terry: I will put it on my todo list. 

Friendly-traceback can currently work (indirectly) with IDLE; see https://aroberge.github.io/friendly-traceback-docs/docs/html/editor.html

I am not sure what benefit this patch will yield when it comes to using Friendly-traceback with IDLE compared with what is currently possible.
msg385560 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 19:12
Andre: with this patch, you could potentially add a function or submodule that would point sys.excepthook to a function that would print the exception and then interact before returning.  If this worked, one could add, for instance, 'import friendly_traceback.idle' at the top of a file or at a Shell prompt and have friendly-traceback only activate when there is an exception.  So it would work in Shell.

I hope by 3.10 to have IDLE '>>>' prompts moved into a sidebar (instead of line numbers.  F-T prompts would be distinctly different.
msg385561 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-23 19:16
Ken, I read on another issue that an exception in an except suite is automatically linked to the caught exception, though the printed result may not be exactly the same.  Once your CLA is 'activated', I will experiment with the options.
msg385719 - (view) Author: Ken Hilton (kenny2python) * Date: 2021-01-26 15:08
CLA registers as signed now.
msg385741 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-26 23:56
New changeset 7a34380ad788886f5ad50d4175ceb2d5715b8cff by Ken in branch 'master':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
https://github.com/python/cpython/commit/7a34380ad788886f5ad50d4175ceb2d5715b8cff
msg385742 - (view) Author: miss-islington (miss-islington) Date: 2021-01-27 00:24
New changeset 68102fb9987338a70d69a0162917866e5710458d by Miss Islington (bot) in branch '3.9':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
https://github.com/python/cpython/commit/68102fb9987338a70d69a0162917866e5710458d
msg385743 - (view) Author: miss-islington (miss-islington) Date: 2021-01-27 00:24
New changeset 6f0346d09f78180886982e070cc92425ada96144 by Miss Islington (bot) in branch '3.8':
bpo-43008: Make IDLE respect sys.excepthook (GH-24302)
https://github.com/python/cpython/commit/6f0346d09f78180886982e070cc92425ada96144
msg385744 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-27 00:27
Ken, for next time -- there is a separate News category for IDLE.
msg385745 - (view) Author: Ken Hilton (kenny2python) * Date: 2021-01-27 00:53
Oops, gotcha. Thanks for doing the tests. I'm a Python contributor! :D
msg385748 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-27 04:37
I hope this is not your last contribution.  Another hint: if one uses issue 12643 or #12643, then the tracker makes the link -- and strikes it out if closed.
msg385924 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-29 18:02
New changeset 11d75ec807f05eff1148c049e38b808d11c23b8a by Terry Jan Reedy in branch 'master':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370)
https://github.com/python/cpython/commit/11d75ec807f05eff1148c049e38b808d11c23b8a
msg385925 - (view) Author: miss-islington (miss-islington) Date: 2021-01-29 18:22
New changeset cf883827496d0fbe2c5fe39e4778cd7525f6f7bc by Miss Islington (bot) in branch '3.8':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370)
https://github.com/python/cpython/commit/cf883827496d0fbe2c5fe39e4778cd7525f6f7bc
msg385926 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-29 18:51
New changeset bf782b2636880dd634281f905ae43b8555450ee2 by Miss Islington (bot) in branch '3.9':
bpo-43008: Add 'Patch by Ken Hilton' (GH-24370) (#24374)
https://github.com/python/cpython/commit/bf782b2636880dd634281f905ae43b8555450ee2
msg386475 - (view) Author: Andre Roberge (aroberge) * Date: 2021-02-04 02:40
Terry: as per your earlier comment, I have tried to use Friendly-traceback with the latest version. While it does set the exception hook correctly, it is not useful. 

Friendly-traceback (FT) needs to have access to the code that was executed. FT's REPL caches the code (similarly to what IPython does, but not CPython) prior to execution so that it can retrieve it correctly, have access to the relevant execution frame, etc.  This is not possible when using IDLE's REPL. In fact, for some types of exceptions, when FT tries to get access to the code, it ends up triggering secondary exceptions.

However, using the method describe in FT's documentation (with relies on FT's own console) still works correctly.
msg386487 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-02-04 17:48
IDLE also caches shell input.  Off topic here, I sent Andre an email about how FT could access it.
History
Date User Action Args
2021-02-04 17:48:45terry.reedysetmessages: + msg386487
2021-02-04 02:40:40arobergesetmessages: + msg386475
2021-01-29 18:51:57terry.reedysetmessages: + msg385926
2021-01-29 18:22:55miss-islingtonsetmessages: + msg385925
2021-01-29 18:02:34terry.reedysetmessages: + msg385924
2021-01-29 18:02:34miss-islingtonsetpull_requests: + pull_request23196
2021-01-29 18:02:25miss-islingtonsetpull_requests: + pull_request23195
2021-01-29 17:32:38terry.reedysetpull_requests: + pull_request23194
2021-01-27 04:37:47terry.reedysetmessages: + msg385748
2021-01-27 00:53:14kenny2pythonsetmessages: + msg385745
2021-01-27 00:27:44terry.reedysetstatus: open -> closed
resolution: fixed
messages: + msg385744

stage: patch review -> resolved
2021-01-27 00:24:39miss-islingtonsetmessages: + msg385743
2021-01-27 00:24:23miss-islingtonsetmessages: + msg385742
2021-01-26 23:56:43miss-islingtonsetpull_requests: + pull_request23167
2021-01-26 23:56:34miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request23166
stage: test needed -> patch review
2021-01-26 23:56:05terry.reedysetmessages: + msg385741
2021-01-26 15:08:51kenny2pythonsetmessages: + msg385719
2021-01-23 19:16:08terry.reedysetmessages: + msg385561
2021-01-23 19:12:55terry.reedysetmessages: + msg385560
2021-01-23 17:14:10arobergesetmessages: + msg385550
2021-01-23 17:09:40terry.reedysetmessages: + msg385548
2021-01-23 17:04:23terry.reedysetnosy: + aroberge
messages: + msg385547
2021-01-23 16:19:32kenny2pythonsetfiles: + bpo-43008.patch.2

messages: + msg385542
2021-01-23 14:31:57terry.reedysettitle: IDLE ignores sys.excepthook -> IDLE ignores sys.excepthook in normal, subprocess mode
stage: test needed
messages: + msg385540
versions: - Python 3.6, Python 3.7
2021-01-23 11:30:46kenny2pythoncreate