classification
Title: IDLE debugger: failure stepping through module loading
Type: behavior Stage: resolved
Components: IDLE Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: terry.reedy Nosy List: brett.cannon, geitda, jcdlr, miss-islington, om364@, terry.reedy
Priority: normal Keywords: patch

Created on 2018-03-13 05:02 by jcdlr, last changed 2021-01-10 07:43 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
PositionalList.py om364@, 2018-04-11 22:50 Linked List module user created
data.py geitda, 2020-04-22 00:10 Minimal example
Pull Requests
URL Status Linked Edit
PR 24183 merged terry.reedy, 2021-01-10 06:35
PR 24184 merged miss-islington, 2021-01-10 07:00
PR 24185 merged miss-islington, 2021-01-10 07:00
Messages (20)
msg313721 - (view) Author: Joshua De La Rosa (jcdlr) Date: 2018-03-13 05:02
Taking my first coding class, so I don't know much about coding or python in general, but I  ran into a problem when using the Debugger function for a homework assignment that neither I nor my professor could make sense of. My program executes successfully without running the Debugger or, in the case that I am running the Debugger, it only raises an error when I "Step" through the imported module that I implemented in another program, rather than just hitting "Go". The error it reports is:
AttributeError: '_ModuleLock' object has no attribute 'name'

Not sure which file to submit, since the I created my own module that is used in the program that raises the error when I step through it with the Debugger mode on.
msg313744 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-03-13 12:26
I cannot do anything from this bare description, as I cannot think of any reason why stepping should introduce an error.  It sounds like there are two files involved.  Try to reduce them to the minimum needed to reproduce the issue and then upload.
msg315203 - (view) Author: (om364@) Date: 2018-04-11 22:50
Traceback (most recent call last):
  File "...\PositionalList.py", line 1, in <module>
    from _DoublyLinkedBase import _DoublyLinkedBase
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 148, in __enter__
  File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "...\Python\Python36\lib\bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "...\Python\Python36\lib\bdb.py", line 66, in dispatch_line
    self.user_line(frame)
  File "...\Python\Python36\lib\idlelib\debugger.py", line 24, in user_line
    self.gui.interaction(message, frame)
AttributeError: '_ModuleLock' object has no attribute 'name'

The file works correctly in PowerShell, but in IDLE debbuger there is a error.
msg315208 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-04-11 23:56
We use 'crash' for, on Window, a process stopping either with no explanation or a 'Your process has stopped box' from Windows.

In any case, PositionalList.py will not run without _DoublyLinkedBase.py, which you did not upload.

The traceback has
  File "...\Python\Python36\lib\bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)

Since sometime last summer, that code line is line 51, indicating that you are using an old release of 3.6, probably 3.6.2 or earlier.  The current bugfix release is 3.6.5.  If possible, upgrade to 3.6.5 and retest.

The traceback does not make much sense to me either.  Neither of the IDLE methods userline and interaction obviously access a .name attribute of anything.  The '_ModuleLock' object is created by importlib._bootstrap.  Searching all issues for '_ModuleLock' got 11 hits.  None are obviously about import and tracing.

To determine whether the problem has anything to do with IDLE, single-step debugging should be repeated with pdb, which is Python's text debugger, and also based on bdb.

Brett, can you tell anything from the multiple importlib._bootstrap lines in the traceback?
msg315228 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-04-12 16:36
Without knowing the exact Python version it's hard to tell as line 59 changed between Python 3.5 and where Python 3.6 is now (FYI, the line as it currently sits in Python 3.6 is https://github.com/python/cpython/blob/e98e3385f2acfc6d98f70f8e66c96b752d003b8f/Lib/importlib/_bootstrap.py#L59). But the double-reporting of the same line is a bit odd.

The best I can think of is that IDLE is requesting the __repr__() of `self` while still executing `__init__()` but before `self.name` is set, triggering an AttributeError (although those lines don't exactly line up with that). Otherwise looking at the code for _ModuleLock suggests this really shouldn't happen as `self.name` is set in `__init__()` and there's no use of `del` or `delattr()` that would cause this.
msg326179 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-09-23 20:32
The problem is not limited to user modules.  In duplicate issue #34609 (not closed), the same traceback was obtained with unittest.  I also got the same with asyncio.  There will not be a problem if the module is already loaded in sys.modules, so that importlib is not invoked.

The traceback is obscured by the fact that the executed importlib is frozen, leaving it traceable, but the code not directly available.  Hence the code is omitted from the debugger display and traceback.  However, the line numbers can be used to find the code within Lib/importlib._bootstrap.py.  Using current master (updated last night), the functions and lines executed by stepping with import unittest are (as expected when reading the code, up to the unexpected exception):

_find_and_load: 980
ModuleLock.__init__: 144, 145
ModuleLock.__enter__: 148
_get_module_lock: 163-168, 170-171, 174: lock = _ModuleLock(name)
_ModuleLock.__init__: 59: self.lock = _thread.allocate_lock()

IDLE's visual debugger has name-value panels for locals, including non-locals, and for globals.  It uses repr to gets value representations.  The locals panel is displayed by default.

Before line 174, 'lock' is bound to None, so before executing line 59, the display is "lock:None\nname:'unittest'".  After line 59, debugger apparently tries to get the repr of the in-process instance.  Since the call in 174 has not completed and should not have rebound 'lock' to the instance, I do not understand why.  (Brett, I now understand what you wrote to be pointing as this puzzle also.)  ppperry, additional light would be appreciated.

Given that debugger does try to get the repr of the instance, both Brett Cannon, here, and (ppperry), on duplicate issue #34609 (now closed), have pointed out that _ModuleLock.__repr__ uses self.name:
        return '_ModuleLock({!r}) at {}'.format(self.name, id(self))

I verified that updating the locals panel is the problem by starting over and turning the panel off until past the the assignment to self.name, at which point, the lock value is "_ModuleLock('unittest') at ...".

Debugger should be prepared for repr to fail, and display something informative.  In the present case, perhaps

repr raised "AttributeError: '_ModuleLock' object has no attribute 'name'"

with a check for whether the exception message contains "'.*' object" (and add such if not present).
msg326757 - (view) Author: (ppperry) Date: 2018-10-01 02:09
Line 59 isn't actually executed; the error comes from the trace event that gets fired before line 59, which is the first `line` event in the frame containing the uninitialized _ModuleLock.
msg336176 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-02-21 02:38
A StackOverflow use got the same _ModuleLock message.
https://stackoverflow.com/questions/54785596/debugger-in-python-freezes-over-own-built-modules
msg366935 - (view) Author: Timothy Geiser (geitda) Date: 2020-04-21 19:14
I wish I could be more helpful than to just pipe up with a "this bug affects me, too," note, but wanted to poke this bug report since it's been dormant for 14 months.
With Python 3.8.2 I tried using the pgpdump module (version 1.5, installed from pip) on Windows 10 and wanted to step through how it worked. As soon as I enabled the debugger in IDLE it stopped working, throwing a very similar stack trace. Here's the MWE:

>>> import pgpdump
>>> <turn on debugger, Locals is checked by default>
[DEBUG ON]
>>> pgpdump.BinaryData(b'')
<focus auto-switches to debug window>
<click Step button>
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    pgpdump.BinaryData(b'')
  File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
    if not data:
  File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
    if not data:
  File "C:\Python38\lib\bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "C:\Python38\lib\bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "C:\Python38\lib\idlelib\debugger.py", line 24, in user_line
    self.gui.interaction(message, frame)
AttributeError: 'BinaryData' object has no attribute 'length'

This error only occurs when the Locals checkbox in the debugging window is checked - it runs as expected as long as Locals is unchecked (this minimum [not]working example throws a "pgpdump.utils.PgpdumpException: no data to parse" error, as it should). A longer example with actual data will run for several steps while Locals is unchecked, but fails with the first Step once Locals is checked. A side note is that the specific error here is that the class BinaryData is being asked about it's 'length', rather than the OP's error of _ModuleLock not having a 'name'
Is there anything else I can do to help fix this, given that I'm not familiar with programming bdb or the IDLE debugger GUI?
msg366947 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-04-21 22:08
Timothy, this is already more helpful than a simple 'me too'.  Can you reduce pgdata.dump to a truly minimal file that exhibits the bug?  and then upload it?  Installing any package into my local copy of the master CPython repository is problemmatical.  What I need is a file that I can download elsewhere, load into IDLE, run from a branch for this issue, and edit and rerun.  From the traceback, it appears that it might be as simple as 

class BinaryData:
    def __init__(self, bd):
        <minimal body>

BinaryData(b'')
msg366959 - (view) Author: Timothy Geiser (geitda) Date: 2020-04-22 00:10
It looks like the IDLE debugger seems to call repr on objects to present in the Locals list, even before they've been properly initialized. If __repr__ needs to refer to variables that don't exist until __init__ is done (and I don't think it's unreasonable for __repr__ to assume that __init__ is indeed finished), the debugger either needs wait until __init__ has completed on any given instance before trying to repr it, or otherwise needs to catch a potentially very wide range of exceptions that might be raised from calling __repr__ so early. I prefer the latter solution, since any buggy code that (effectively) crashes on it's __repr__ (for whatever reason) will probably bring the debugger to it's knees.
I played around with pdb directly and can sort of get the same thing if I ask for __repr__ too early:

 1 Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
 2 Type "help", "copyright", "credits" or "license" for more information.
 3 >>> import data
 4 >>> import pdb
 5 >>> pdb.run("data.BinaryData(b'some data')")
 6 > <string>(1)<module>()
 7 (Pdb) step
 8 --Call--
 9 > c:\users\geitd\documents\python\data.py(4)__init__()
10 -> def __init__(self, data):
11 (Pdb) p self
12 (Pdb) p BinaryData.__repr__(self)
13 *** AttributeError: 'BinaryData' object has no attribute 'length'
14 (Pdb) step
15 > c:\users\geitd\documents\python\data.py(5)__init__()
16 -> if not data:
17 (Pdb) step
18 > c:\users\geitd\documents\python\data.py(7)__init__()
19 -> if len(data) <= 1:
20 (Pdb) step
21 > c:\users\geitd\documents\python\data.py(10)__init__()
22 -> self.data = data
23 (Pdb) step
24 > c:\users\geitd\documents\python\data.py(11)__init__()
25 -> self.length = len(data)
26 (Pdb) p self
27 (Pdb) p BinaryData.__repr__(self)
28 *** AttributeError: 'BinaryData' object has no attribute 'length'
29 (Pdb) step
30 --Return--
31 > c:\users\geitd\documents\python\data.py(11)__init__()->None
32 -> self.length = len(data)
33 (Pdb) p self
34 <BinaryData: length 9>
35 (Pdb) p BinaryData.__repr__(self)
36 '<BinaryData: length 9>'

Note that line 11 didn't return anything, but didn't have any bad results, whereas the way I phrased line 12 gave the exact same error the IDLE debugger threw.
Lines 26 and 27 towards the end of __init__ came out the same, but after the --Return-- on 30, either phrasing gives what you'd expect.
I suppose the TL;DR is to take the mechanism that gives the correct behavior of 'p self' in pdb and copy it over to the IDLE debugger (or whatever other mechanism is necessary).

Is this enough for you to work from?
msg366981 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-04-22 06:58
I believe I found the bug.  For IDLE's original single process mode, still available with the -n startup option, debugger.py contains the entire debugger.  The values displayed for global and local names are obtained with reprlib.Repr.repr.  That in turn calls .repr1, and that calls .repr_xxx, where xxx is one of the 'common' builtin classes or 'instance'.

The latter is used for all user classes.  It calls __builtins__.repr, but guarded by 'try...except Exception' since user classes may cause exceptions.  The except clause returns an alternative type and id string, like object.__repr__.  (That alternative could also raise, but much less often.  Any of the examples above should run if IDLE were started from a command line with 'python -m idlelib -n'.

When user code is run in a separate process, the code that interacts with user object must also run in the separate process.  debugger.Idb is moved and the code in debugger_r is added, some in each process.  Of concern here is that the GUI code that displays global or local values is passed a dict proxy instead of an actual namespace dict.  The proxy __getitem__ for d[key] makes an rpc call to through the socket connection to code in the user process.  That returns not the object itself but a string representation.  It does so with an unguarded repr call.

IDLE intentionally removes traceback lines added by IDLE (and pdb, if used), so that tracebacks look mostly the same as in standard CPython.  But that is a handicap when there is a bug in IDLE.  A traceback ending with 
  File .../idlelib/debugger_r, line 173, in dict_item
    value = repr(value)
AttributeError: ...

would have been a big help here.  I am thinking about how to selectively disable traceback cleanup.

In any case, I believe the solution is to import reprlib in debugger_r also and add 'reprlib.Repr' before 'repr' in the line above.
msg367149 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-04-23 21:31
Timothy, can you try editing idlelib.debugger_r, line 173, as suggested above and see if it solves the problem?
msg367152 - (view) Author: Timothy Geiser (geitda) Date: 2020-04-23 22:06
Looks good when testing both the minimal example and the pgpdump original case.

I added the import at the top, and changed the original line 173 from

value = repr(value)

to

value = reprlib.repr(value)

This is based on lines 160 & 161 in reprlib (we need a Repr instance, but reprlib makes it's own for us to use).

I get a nice boring and not-crashy result in the debug window for Locals:

data   b''
self   <BinaryData instance at 0x2726a0ab880>

Thanks for the fix, Terry!
I shouldn't expect any strange side-effects from this, should I?
msg369341 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2020-05-19 12:22
SO duplicate.
https://stackoverflow.com/questions/61310989/python-idle-importing-xlrd-error-generated-in-debug-mode-attributeerror-mo
msg384742 - (view) Author: Timothy Geiser (geitda) Date: 2021-01-09 22:45
This issue is still open, 8.5 months after identifying the underlying cause.
What needs done to get a fix for this merged? Manual testing, or adding test cases? Anything I can do to help? I have 3.9.1 successfully built on a Raspberry Pi that I can reproduce the issue on, as well as reproduce the fix Terry suggested in msg366981.
msg384752 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-10 06:39
I am reluctant to make changes without adding tests, and until I focused again on how to do so, because of your ping, I had no idea how to do so.

The added test results in the same error, "AttributeError: 'BinData' object has no attribute 'length'" without the patch and passes with it.
msg384753 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-01-10 06:59
New changeset 81f87bbf9f65702062021a78abd9b8f82c98a414 by Terry Jan Reedy in branch 'master':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/81f87bbf9f65702062021a78abd9b8f82c98a414
msg384754 - (view) Author: miss-islington (miss-islington) Date: 2021-01-10 07:31
New changeset 799f8489d418b7f9207d333eac38214931bd7dcc by Miss Islington (bot) in branch '3.9':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/799f8489d418b7f9207d333eac38214931bd7dcc
msg384755 - (view) Author: miss-islington (miss-islington) Date: 2021-01-10 07:31
New changeset 5ded7efa6a7a232dd4a41e6e65e4dae47146514b by Miss Islington (bot) in branch '3.8':
bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
https://github.com/python/cpython/commit/5ded7efa6a7a232dd4a41e6e65e4dae47146514b
History
Date User Action Args
2021-01-10 07:43:30terry.reedysetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-01-10 07:31:11miss-islingtonsetmessages: + msg384755
2021-01-10 07:31:11miss-islingtonsetmessages: + msg384754
2021-01-10 07:00:38miss-islingtonsetpull_requests: + pull_request23013
2021-01-10 07:00:28miss-islingtonsetnosy: + miss-islington

pull_requests: + pull_request23012
stage: commit review -> patch review
2021-01-10 06:59:54terry.reedysetmessages: + msg384753
2021-01-10 06:39:17terry.reedysetstage: patch review -> commit review
messages: + msg384752
versions: + Python 3.10, - Python 3.7
2021-01-10 06:35:11terry.reedysetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request23011
2021-01-09 22:45:17geitdasetmessages: + msg384742
2020-05-19 12:22:33terry.reedysetmessages: + msg369341
2020-04-23 22:06:07geitdasetmessages: + msg367152
2020-04-23 21:51:20ppperrysetnosy: - ppperry
2020-04-23 21:31:22terry.reedysetmessages: + msg367149
2020-04-22 06:58:50terry.reedysetmessages: + msg366981
versions: + Python 3.9
2020-04-22 00:10:28geitdasetfiles: + data.py

messages: + msg366959
2020-04-21 22:08:38terry.reedysetmessages: + msg366947
2020-04-21 19:14:11geitdasetnosy: + geitda
messages: + msg366935
2019-02-21 02:38:09terry.reedysetmessages: + msg336176
2018-12-11 22:31:52terry.reedysetversions: - Python 3.6
2018-10-01 02:09:04ppperrysetnosy: + ppperry
messages: + msg326757
2018-09-23 20:32:27terry.reedysetnosy: - ppperry
title: IDLE debugger crashes when `repr` raises an exception -> IDLE debugger: failure stepping through module loading
messages: + msg326179

versions: + Python 3.7, Python 3.8
2018-09-23 18:29:35ppperrysettitle: IDLE debugger: problem importing user created module -> IDLE debugger crashes when `repr` raises an exception
2018-09-23 18:23:32ppperrysetnosy: + ppperry
2018-09-23 18:14:03terry.reedylinkissue34609 superseder
2018-04-12 16:36:04brett.cannonsetmessages: + msg315228
2018-04-11 23:56:10terry.reedysetnosy: + brett.cannon
type: crash -> behavior
messages: + msg315208
2018-04-11 22:50:54om364@setfiles: + PositionalList.py

nosy: + om364@
messages: + msg315203

type: behavior -> crash
2018-03-13 12:26:02terry.reedysettype: compile error -> behavior
title: debugger issue concerning importing user created modules into another program -> IDLE debugger: problem importing user created module
messages: + msg313744
stage: test needed
2018-03-13 05:02:16jcdlrcreate