Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IDLE debugger: failure stepping through module loading #77246

Closed
jcdlr mannequin opened this issue Mar 13, 2018 · 20 comments
Closed

IDLE debugger: failure stepping through module loading #77246

jcdlr mannequin opened this issue Mar 13, 2018 · 20 comments
Assignees
Labels
3.8 only security fixes 3.9 only security fixes 3.10 only security fixes topic-IDLE type-bug An unexpected behavior, bug, or error

Comments

@jcdlr
Copy link
Mannequin

jcdlr mannequin commented Mar 13, 2018

BPO 33065
Nosy @brettcannon, @terryjreedy, @miss-islington
PRs
  • bpo-33065: Fix problem debugging user classes with __repr__ method #24183
  • [3.9] bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183) #24184
  • [3.8] bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183) #24185
  • Files
  • PositionalList.py: Linked List module user created
  • data.py: Minimal example
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/terryjreedy'
    closed_at = <Date 2021-01-10.07:43:30.003>
    created_at = <Date 2018-03-13.05:02:16.611>
    labels = ['expert-IDLE', 'type-bug', '3.8', '3.9', '3.10']
    title = 'IDLE debugger: failure stepping through module loading'
    updated_at = <Date 2021-01-10.07:43:30.003>
    user = 'https://bugs.python.org/jcdlr'

    bugs.python.org fields:

    activity = <Date 2021-01-10.07:43:30.003>
    actor = 'terry.reedy'
    assignee = 'terry.reedy'
    closed = True
    closed_date = <Date 2021-01-10.07:43:30.003>
    closer = 'terry.reedy'
    components = ['IDLE']
    creation = <Date 2018-03-13.05:02:16.611>
    creator = 'jcdlr'
    dependencies = []
    files = ['47532', '49082']
    hgrepos = []
    issue_num = 33065
    keywords = ['patch']
    message_count = 20.0
    messages = ['313721', '313744', '315203', '315208', '315228', '326179', '326757', '336176', '366935', '366947', '366959', '366981', '367149', '367152', '369341', '384742', '384752', '384753', '384754', '384755']
    nosy_count = 6.0
    nosy_names = ['brett.cannon', 'terry.reedy', 'geitda', 'miss-islington', 'jcdlr', 'om364@']
    pr_nums = ['24183', '24184', '24185']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue33065'
    versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

    @jcdlr
    Copy link
    Mannequin Author

    jcdlr mannequin commented Mar 13, 2018

    Taking my first coding class, so I don't know much about coding or python in general, but I ran into a problem when using the Debugger function for a homework assignment that neither I nor my professor could make sense of. My program executes successfully without running the Debugger or, in the case that I am running the Debugger, it only raises an error when I "Step" through the imported module that I implemented in another program, rather than just hitting "Go". The error it reports is:
    AttributeError: '_ModuleLock' object has no attribute 'name'

    Not sure which file to submit, since the I created my own module that is used in the program that raises the error when I step through it with the Debugger mode on.

    @jcdlr jcdlr mannequin added the build The build process and cross-build label Mar 13, 2018
    @jcdlr jcdlr mannequin assigned terryjreedy Mar 13, 2018
    @jcdlr jcdlr mannequin added the topic-IDLE label Mar 13, 2018
    @terryjreedy
    Copy link
    Member

    I cannot do anything from this bare description, as I cannot think of any reason why stepping should introduce an error. It sounds like there are two files involved. Try to reduce them to the minimum needed to reproduce the issue and then upload.

    @terryjreedy terryjreedy added type-bug An unexpected behavior, bug, or error and removed build The build process and cross-build labels Mar 13, 2018
    @terryjreedy terryjreedy changed the title debugger issue concerning importing user created modules into another program IDLE debugger: problem importing user created module Mar 13, 2018
    @om364
    Copy link
    Mannequin

    om364 mannequin commented Apr 11, 2018

    Traceback (most recent call last):
      File "...\PositionalList.py", line 1, in <module>
        from _DoublyLinkedBase import _DoublyLinkedBase
      File "<frozen importlib._bootstrap>", line 968, in _find_and_load
      File "<frozen importlib._bootstrap>", line 148, in __enter__
      File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
      File "<frozen importlib._bootstrap>", line 59, in __init__
      File "<frozen importlib._bootstrap>", line 59, in __init__
      File "...\Python\Python36\lib\bdb.py", line 48, in trace_dispatch
        return self.dispatch_line(frame)
      File "...\Python\Python36\lib\bdb.py", line 66, in dispatch_line
        self.user_line(frame)
      File "...\Python\Python36\lib\idlelib\debugger.py", line 24, in user_line
        self.gui.interaction(message, frame)
    AttributeError: '_ModuleLock' object has no attribute 'name'

    The file works correctly in PowerShell, but in IDLE debbuger there is a error.

    @om364 om364 mannequin added type-crash A hard crash of the interpreter, possibly with a core dump and removed type-bug An unexpected behavior, bug, or error labels Apr 11, 2018
    @terryjreedy
    Copy link
    Member

    We use 'crash' for, on Window, a process stopping either with no explanation or a 'Your process has stopped box' from Windows.

    In any case, PositionalList.py will not run without _DoublyLinkedBase.py, which you did not upload.

    The traceback has
    File "...\Python\Python36\lib\bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)

    Since sometime last summer, that code line is line 51, indicating that you are using an old release of 3.6, probably 3.6.2 or earlier. The current bugfix release is 3.6.5. If possible, upgrade to 3.6.5 and retest.

    The traceback does not make much sense to me either. Neither of the IDLE methods userline and interaction obviously access a .name attribute of anything. The '_ModuleLock' object is created by importlib._bootstrap. Searching all issues for '_ModuleLock' got 11 hits. None are obviously about import and tracing.

    To determine whether the problem has anything to do with IDLE, single-step debugging should be repeated with pdb, which is Python's text debugger, and also based on bdb.

    Brett, can you tell anything from the multiple importlib._bootstrap lines in the traceback?

    @terryjreedy terryjreedy added type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Apr 11, 2018
    @brettcannon
    Copy link
    Member

    Without knowing the exact Python version it's hard to tell as line 59 changed between Python 3.5 and where Python 3.6 is now (FYI, the line as it currently sits in Python 3.6 is

    self.lock = _thread.allocate_lock()
    ). But the double-reporting of the same line is a bit odd.

    The best I can think of is that IDLE is requesting the repr() of self while still executing __init__() but before self.name is set, triggering an AttributeError (although those lines don't exactly line up with that). Otherwise looking at the code for _ModuleLock suggests this really shouldn't happen as self.name is set in __init__() and there's no use of del or delattr() that would cause this.

    @pppery pppery mannequin changed the title IDLE debugger: problem importing user created module IDLE debugger crashes when repr raises an exception Sep 23, 2018
    @terryjreedy
    Copy link
    Member

    The problem is not limited to user modules. In duplicate issue bpo-34609 (not closed), the same traceback was obtained with unittest. I also got the same with asyncio. There will not be a problem if the module is already loaded in sys.modules, so that importlib is not invoked.

    The traceback is obscured by the fact that the executed importlib is frozen, leaving it traceable, but the code not directly available. Hence the code is omitted from the debugger display and traceback. However, the line numbers can be used to find the code within Lib/importlib._bootstrap.py. Using current master (updated last night), the functions and lines executed by stepping with import unittest are (as expected when reading the code, up to the unexpected exception):

    _find_and_load: 980
    ModuleLock.__init__: 144, 145
    ModuleLock.__enter__: 148
    _get_module_lock: 163-168, 170-171, 174: lock = _ModuleLock(name)
    _ModuleLock.__init__: 59: self.lock = _thread.allocate_lock()

    IDLE's visual debugger has name-value panels for locals, including non-locals, and for globals. It uses repr to gets value representations. The locals panel is displayed by default.

    Before line 174, 'lock' is bound to None, so before executing line 59, the display is "lock:None\nname:'unittest'". After line 59, debugger apparently tries to get the repr of the in-process instance. Since the call in 174 has not completed and should not have rebound 'lock' to the instance, I do not understand why. (Brett, I now understand what you wrote to be pointing as this puzzle also.) ppperry, additional light would be appreciated.

    Given that debugger does try to get the repr of the instance, both Brett Cannon, here, and (ppperry), on duplicate issue bpo-34609 (now closed), have pointed out that _ModuleLock.__repr__ uses self.name:
    return '_ModuleLock({!r}) at {}'.format(self.name, id(self))

    I verified that updating the locals panel is the problem by starting over and turning the panel off until past the the assignment to self.name, at which point, the lock value is "_ModuleLock('unittest') at ...".

    Debugger should be prepared for repr to fail, and display something informative. In the present case, perhaps

    repr raised "AttributeError: '_ModuleLock' object has no attribute 'name'"

    with a check for whether the exception message contains "'.*' object" (and add such if not present).

    @terryjreedy terryjreedy added 3.7 (EOL) end of life 3.8 only security fixes labels Sep 23, 2018
    @terryjreedy terryjreedy changed the title IDLE debugger crashes when repr raises an exception IDLE debugger: failure stepping through module loading Sep 23, 2018
    @pppery
    Copy link
    Mannequin

    pppery mannequin commented Oct 1, 2018

    Line 59 isn't actually executed; the error comes from the trace event that gets fired before line 59, which is the first line event in the frame containing the uninitialized _ModuleLock.

    @terryjreedy
    Copy link
    Member

    A StackOverflow use got the same _ModuleLock message.
    https://stackoverflow.com/questions/54785596/debugger-in-python-freezes-over-own-built-modules

    @geitda
    Copy link
    Mannequin

    geitda mannequin commented Apr 21, 2020

    I wish I could be more helpful than to just pipe up with a "this bug affects me, too," note, but wanted to poke this bug report since it's been dormant for 14 months.
    With Python 3.8.2 I tried using the pgpdump module (version 1.5, installed from pip) on Windows 10 and wanted to step through how it worked. As soon as I enabled the debugger in IDLE it stopped working, throwing a very similar stack trace. Here's the MWE:

    >>> import pgpdump
    >>> <turn on debugger, Locals is checked by default>
    [DEBUG ON]
    >>> pgpdump.BinaryData(b'')
    <focus auto-switches to debug window>
    <click Step button>
    Traceback (most recent call last):
      File "<pyshell#1>", line 1, in <module>
        pgpdump.BinaryData(b'')
      File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
        if not data:
      File "C:\Python38\lib\site-packages\pgpdump\data.py", line 13, in __init__
        if not data:
      File "C:\Python38\lib\bdb.py", line 88, in trace_dispatch
        return self.dispatch_line(frame)
      File "C:\Python38\lib\bdb.py", line 112, in dispatch_line
        self.user_line(frame)
      File "C:\Python38\lib\idlelib\debugger.py", line 24, in user_line
        self.gui.interaction(message, frame)
    AttributeError: 'BinaryData' object has no attribute 'length'

    This error only occurs when the Locals checkbox in the debugging window is checked - it runs as expected as long as Locals is unchecked (this minimum [not]working example throws a "pgpdump.utils.PgpdumpException: no data to parse" error, as it should). A longer example with actual data will run for several steps while Locals is unchecked, but fails with the first Step once Locals is checked. A side note is that the specific error here is that the class BinaryData is being asked about it's 'length', rather than the OP's error of _ModuleLock not having a 'name'
    Is there anything else I can do to help fix this, given that I'm not familiar with programming bdb or the IDLE debugger GUI?

    @terryjreedy
    Copy link
    Member

    Timothy, this is already more helpful than a simple 'me too'. Can you reduce pgdata.dump to a truly minimal file that exhibits the bug? and then upload it? Installing any package into my local copy of the master CPython repository is problemmatical. What I need is a file that I can download elsewhere, load into IDLE, run from a branch for this issue, and edit and rerun. From the traceback, it appears that it might be as simple as

    class BinaryData:
        def __init__(self, bd):
            <minimal body>
    
    BinaryData(b'')

    @geitda
    Copy link
    Mannequin

    geitda mannequin commented Apr 22, 2020

    It looks like the IDLE debugger seems to call repr on objects to present in the Locals list, even before they've been properly initialized. If __repr__ needs to refer to variables that don't exist until __init__ is done (and I don't think it's unreasonable for __repr__ to assume that __init__ is indeed finished), the debugger either needs wait until __init__ has completed on any given instance before trying to repr it, or otherwise needs to catch a potentially very wide range of exceptions that might be raised from calling __repr__ so early. I prefer the latter solution, since any buggy code that (effectively) crashes on it's __repr__ (for whatever reason) will probably bring the debugger to it's knees.
    I played around with pdb directly and can sort of get the same thing if I ask for __repr__ too early:

     1 Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32
     2 Type "help", "copyright", "credits" or "license" for more information.
     3 >>> import data
     4 >>> import pdb
     5 >>> pdb.run("data.BinaryData(b'some data')")
     6 > <string>(1)<module>()
     7 (Pdb) step
     8 --Call--
     9 > c:\users\geitd\documents\python\data.py(4)__init__()
    10 -> def __init__(self, data):
    11 (Pdb) p self
    12 (Pdb) p BinaryData.__repr__(self)
    13 *** AttributeError: 'BinaryData' object has no attribute 'length'
    14 (Pdb) step
    15 > c:\users\geitd\documents\python\data.py(5)__init__()
    16 -> if not data:
    17 (Pdb) step
    18 > c:\users\geitd\documents\python\data.py(7)__init__()
    19 -> if len(data) <= 1:
    20 (Pdb) step
    21 > c:\users\geitd\documents\python\data.py(10)__init__()
    22 -> self.data = data
    23 (Pdb) step
    24 > c:\users\geitd\documents\python\data.py(11)__init__()
    25 -> self.length = len(data)
    26 (Pdb) p self
    27 (Pdb) p BinaryData.__repr__(self)
    28 *** AttributeError: 'BinaryData' object has no attribute 'length'
    29 (Pdb) step
    30 --Return--
    31 > c:\users\geitd\documents\python\data.py(11)__init__()->None
    32 -> self.length = len(data)
    33 (Pdb) p self
    34 <BinaryData: length 9>
    35 (Pdb) p BinaryData.__repr__(self)
    36 '<BinaryData: length 9>'

    Note that line 11 didn't return anything, but didn't have any bad results, whereas the way I phrased line 12 gave the exact same error the IDLE debugger threw.
    Lines 26 and 27 towards the end of __init__ came out the same, but after the --Return-- on 30, either phrasing gives what you'd expect.
    I suppose the TL;DR is to take the mechanism that gives the correct behavior of 'p self' in pdb and copy it over to the IDLE debugger (or whatever other mechanism is necessary).

    Is this enough for you to work from?

    @terryjreedy
    Copy link
    Member

    I believe I found the bug. For IDLE's original single process mode, still available with the -n startup option, debugger.py contains the entire debugger. The values displayed for global and local names are obtained with reprlib.Repr.repr. That in turn calls .repr1, and that calls .repr_xxx, where xxx is one of the 'common' builtin classes or 'instance'.

    The latter is used for all user classes. It calls __builtins__.repr, but guarded by 'try...except Exception' since user classes may cause exceptions. The except clause returns an alternative type and id string, like object.__repr__. (That alternative could also raise, but much less often. Any of the examples above should run if IDLE were started from a command line with 'python -m idlelib -n'.

    When user code is run in a separate process, the code that interacts with user object must also run in the separate process. debugger.Idb is moved and the code in debugger_r is added, some in each process. Of concern here is that the GUI code that displays global or local values is passed a dict proxy instead of an actual namespace dict. The proxy __getitem__ for d[key] makes an rpc call to through the socket connection to code in the user process. That returns not the object itself but a string representation. It does so with an unguarded repr call.

    IDLE intentionally removes traceback lines added by IDLE (and pdb, if used), so that tracebacks look mostly the same as in standard CPython. But that is a handicap when there is a bug in IDLE. A traceback ending with
    File .../idlelib/debugger_r, line 173, in dict_item
    value = repr(value)
    AttributeError: ...

    would have been a big help here. I am thinking about how to selectively disable traceback cleanup.

    In any case, I believe the solution is to import reprlib in debugger_r also and add 'reprlib.Repr' before 'repr' in the line above.

    @terryjreedy terryjreedy added the 3.9 only security fixes label Apr 22, 2020
    @terryjreedy
    Copy link
    Member

    Timothy, can you try editing idlelib.debugger_r, line 173, as suggested above and see if it solves the problem?

    @geitda
    Copy link
    Mannequin

    geitda mannequin commented Apr 23, 2020

    Looks good when testing both the minimal example and the pgpdump original case.

    I added the import at the top, and changed the original line 173 from

    value = repr(value)

    to

    value = reprlib.repr(value)

    This is based on lines 160 & 161 in reprlib (we need a Repr instance, but reprlib makes it's own for us to use).

    I get a nice boring and not-crashy result in the debug window for Locals:

    data b''
    self <BinaryData instance at 0x2726a0ab880>

    Thanks for the fix, Terry!
    I shouldn't expect any strange side-effects from this, should I?

    @geitda
    Copy link
    Mannequin

    geitda mannequin commented Jan 9, 2021

    This issue is still open, 8.5 months after identifying the underlying cause.
    What needs done to get a fix for this merged? Manual testing, or adding test cases? Anything I can do to help? I have 3.9.1 successfully built on a Raspberry Pi that I can reproduce the issue on, as well as reproduce the fix Terry suggested in msg366981.

    @terryjreedy
    Copy link
    Member

    I am reluctant to make changes without adding tests, and until I focused again on how to do so, because of your ping, I had no idea how to do so.

    The added test results in the same error, "AttributeError: 'BinData' object has no attribute 'length'" without the patch and passes with it.

    @terryjreedy terryjreedy added 3.10 only security fixes and removed 3.7 (EOL) end of life labels Jan 10, 2021
    @terryjreedy
    Copy link
    Member

    New changeset 81f87bb by Terry Jan Reedy in branch 'master':
    bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
    81f87bb

    @miss-islington
    Copy link
    Contributor

    New changeset 799f848 by Miss Islington (bot) in branch '3.9':
    bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
    799f848

    @miss-islington
    Copy link
    Contributor

    New changeset 5ded7ef by Miss Islington (bot) in branch '3.8':
    bpo-33065: Fix problem debugging user classes with __repr__ method (GH-24183)
    5ded7ef

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes 3.9 only security fixes 3.10 only security fixes topic-IDLE type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants