This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: IDLE removes elements from tracebacks.
Type: behavior Stage: patch review
Components: IDLE Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: kbk, ppperry, roger.serwy, terry.reedy
Priority: normal Keywords: patch

Created on 2015-05-21 00:44 by ppperry, last changed 2022-04-11 14:58 by admin.

Files
File name Uploaded Description Edit
tb-trim.diff terry.reedy, 2015-06-02 23:32 still needs comment added review
Messages (10)
msg243709 - (view) Author: (ppperry) Date: 2015-05-21 00:44
When reproducing issue24160 in IDLE, I saw this traceback:
Traceback (most recent call last):
  File "<pyshell#14>", line 1, in <module>
    pdb.run("reload(test3)")
  File "C:\Python27\lib\pdb.py", line 1238, in run
    Pdb().run(statement, globals, locals)
  File "C:\Python27\lib\bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "test3.py", line 1, in <module>
    [contents of test3.py]
  File "C:\Python27\lib\bdb.py", line 51, in trace_dispatch
    return self.dispatch_call(frame, arg)
  File "C:\Python27\lib\bdb.py", line 80, in dispatch_call
    self.user_call(frame, arg)
  File "C:\Python27\lib\pdb.py", line 148, in user_call
    self.interaction(frame, None)
  File "C:\Python27\lib\pdb.py", line 210, in interaction
    self.cmdloop()
  File "C:\Python27\lib\cmd.py", line 142, in cmdloop
    stop = self.onecmd(line)
  File "C:\Python27\lib\pdb.py", line 279, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "C:\Python27\lib\cmd.py", line 221, in onecmd
    return func(arg)
  File "C:\Python27\lib\pdb.py", line 622, in do_clear
    err = self.clear_bpbynumber(i)
KeyError: 'c:\\documents and settings\\perry\\desktop\\coding_projects\\python\\test3.py'

Reproducing the same bug in the standard interpreter produces a more detailed traceback:
Traceback (most recent call last):
  File "<pyshell#592>", line 1, in <module>
    pdb.run("reload(test3)")
  File "C:\Python27\lib\pdb.py", line 1238, in run
    Pdb().run(statement, globals, locals)
  File "C:\Python27\lib\bdb.py", line 400, in run
    exec cmd in globals, locals
  File "<string>", line 1, in <module>
  File "test3.py", line 1, in <module>
    foo = 7789
  File "C:\Python27\lib\bdb.py", line 51, in trace_dispatch
    return self.dispatch_call(frame, arg)
  File "C:\Python27\lib\bdb.py", line 80, in dispatch_call
    self.user_call(frame, arg)
  File "C:\Python27\lib\pdb.py", line 148, in user_call
    self.interaction(frame, None)
  File "C:\Python27\lib\pdb.py", line 210, in interaction
    self.cmdloop()
  File "C:\Python27\lib\cmd.py", line 142, in cmdloop
    stop = self.onecmd(line)
  File "C:\Python27\lib\pdb.py", line 279, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "C:\Python27\lib\cmd.py", line 221, in onecmd
    return func(arg)
  File "C:\Python27\lib\pdb.py", line 622, in do_clear
    err = self.clear_bpbynumber(i)
  File "C:\Python27\lib\bdb.py", line 297, in clear_bpbynumber
    self._prune_breaks(bp.file, bp.line)
  File "C:\Python27\lib\bdb.py", line 268, in _prune_breaks
    self.breaks[filename].remove(lineno)
KeyError: 'c:\\documents and settings\\perry\\desktop\\coding_projects\\python\\test3.py'

The actual error message is irrelevant, other than the fact that in the standard interpreter, the traceback contains two more entries.
msg243737 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-05-21 07:02
The two additional entries are the last two.  In this case, they are needed because KeyError pertains to the dict lookup in the last line "self.breaks[filename]".

Mistaken in this particular case or not, the deletion is intentional. Run.print_exception has these two lines:
    exclude = ("run.py", "rpc.py", "threading.py", "Queue.py",
               "RemoteDebugger.py", "bdb.py")
    cleanup_traceback(tbe, exclude)
The docstring for cleanup_traceback is "Remove excluded traces from beginning/end of tb; get cached lines".

The intent of the first 4 excludes is to make the tracebacks displayed in Idle for normal (non-debug) code the same as with Python, without Idle.  The intent of the last 2 is to continue the equivalence when using the Idle debugger (which inherits from bdb.py).

Using pdb with Idle is known to be somewhat problematical because Idle revises the execution environment.  In particular, identical tracebacks cannot be guaranteed.  (I tried to find the issue bug the buggy tracker search did not return the issue.)

A fix for this particular example would be to omit/remove bdb from exclude (which could be a set) when the first item contains 'pdb'.  I would consider a patch to do that.
msg244700 - (view) Author: (ppperry) Date: 2015-06-02 22:01
This problem also occurs in other situati, such as when trying to get items from an empty queue
>>> import Queue
>>> Q = Queue.Queue()
>>> q.get_nowait()
Traceback (most recent call last):
  File "<pyshell#16>", line 1, in <module>
    q.get_nowait()
Empty
In that case, it doesn't really matter, though.
msg244714 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-06-02 23:32
Your message arrived as I was just starting on this issue, wondering whether the exclusion of threading and queue blocks is proper and wishing for an easy test example, such as you provided. It verifies the issue in 3.x.  It also answers the question: trimming queue blocks is not acceptible.  (Many threading functions also raise various exceptions.) Informative tracebacks are a major python feature.  So Idle should be conservative about trimming tracebacks.

The attached 3.4 patch solves queue and threading problems by deleting them from the exclude list.  It attempts to solve your pdb problem by conditionally removing bdb.py also.  Could you patch your Idle, by hand if necessary, and test?
msg244932 - (view) Author: (ppperry) Date: 2015-06-06 23:29
Is there any reason why the end of the traceback, rather then just the beginning, needs to be pruned in the first place?

Additionally, the "search for pdb in the tb" method will still undesirably prune the traceback if someone invents there own buggy debugger that subclasses from bdb.Bdb.

Would this cleanup function work?
def cleanup_traceback(tb):
    idle_directory = os.path.dirname(__file__)
    while os.path.dirname(tb[0][0]) == idle_directory:
        del tb[0]
    [rest of code not having to do with pruning tracebacks]
The only situation that this function would behave wrongly is when someone launches IDLE from the shell (Why would they do that?)
msg244935 - (view) Author: (ppperry) Date: 2015-06-07 00:09
Another example of this overzealous removing is when you create a module named rpc, run, RemoteDebugger, or bdb. 
For example (in this environment, a file in the current directory named rpc.py exists and refers to the undefined name "bar"):
>>>import rpc
Traceback (most recent call last):
  File "<pyshell#x>", line 1, in <module>
    import rpc
NameError: name 'bar' is not defined

The correct traceback is:
Traceback (most recent call last):
  File "<pyshell#x>", line 1, in <module>
  File "rpc.py", line 2, in <module>
    bar
NameError: name 'bar' is not defined

The IDLE debugger also refuses to debug code in the above file (only if it is named "rpc" not any of the other names).
msg244985 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-06-08 04:25
Did you test my proposed patch?  I am reluctant to make major changes in the absence of either documentation for why the code is as it is or test cases.  So I do not know about trimming top versus bottom and the examples intended to show both are needed.  I am not even sure that only top and bottom are 'correct'.

That said, paying attention to the complete file name, or at least the prior directory ('idlelib' for idle files), seems plausible.  But we have to attend to '/' versus Windows' '\'.

"The only situation that this function would behave wrongly is when someone launches IDLE from the shell (Why would they do that?)"

Which shell?  I sometimes start Idle from both command line shell and python shell, for different reasons.

I was already aware that pdb and Idle might not be the only things to subclass bdb (with bugs). I think the patch should be 'inverted' to only add bdb to the exclude list if idlelib/RemoteDebugger.py (or ...\...) is present in the traceback. 

The current code was moved from Executive.runcode 2003-05-08 by KBK.
"2. run.py: move exception printing to toplevel to allow access from main()"
A couple of months before, the exclude list was shorter:
   exclude = ("run.py", "rpc.py", "RemoteDebugger.py", "bdb.py")
I did not find the reason for adding threading and queue.
A little before, the exception printing code was in rpc.py. I will do more history tracing later.
msg244999 - (view) Author: (ppperry) Date: 2015-06-08 11:40
Your proposed patch does not work in its current form on my IDLE, but it does if I change tr[0][:-6] to tr[0][-6:] in the pdb checking code. 

Additionally, my proposed cleanup function did distinguish the difference between the unix and windows directory seperators by using os.path.dirname.
msg245000 - (view) Author: (ppperry) Date: 2015-06-08 11:41
I mean if someone starts a new copy of idle from within the IDLE python shell itself by "The only situation that this function would behave wrongly is when someone launches IDLE from the shell (Why would they do that?)".
msg296672 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-23 04:24
#26627 was closed as an apparent duplicate of this.  It should be rechecked if this is fixed.
History
Date User Action Args
2022-04-11 14:58:17adminsetgithub: 68440
2020-06-07 22:06:34terry.reedysetversions: + Python 3.10, - Python 3.6, Python 3.7
2017-06-23 04:24:35terry.reedysetassignee: terry.reedy
messages: + msg296672
versions: + Python 3.7, - Python 2.7, Python 3.4, Python 3.5
2017-06-23 04:23:17terry.reedylinkissue26627 superseder
2015-06-08 11:41:35ppperrysetmessages: + msg245000
2015-06-08 11:40:20ppperrysetmessages: + msg244999
2015-06-08 04:25:18terry.reedysetmessages: + msg244985
2015-06-07 00:09:09ppperrysetmessages: + msg244935
2015-06-06 23:29:21ppperrysetmessages: + msg244932
2015-06-02 23:32:07terry.reedysetfiles: + tb-trim.diff
versions: + Python 3.4, Python 3.5, Python 3.6
messages: + msg244714

keywords: + patch
stage: needs patch -> patch review
2015-06-02 22:01:55ppperrysetmessages: + msg244700
2015-05-21 07:02:56terry.reedysetmessages: + msg243737
stage: needs patch
2015-05-21 00:44:08ppperrycreate