Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importing certain modules while debugging raises an exception #78790

Closed
piscvauyahoofr mannequin opened this issue Sep 7, 2018 · 11 comments
Closed

Importing certain modules while debugging raises an exception #78790

piscvauyahoofr mannequin opened this issue Sep 7, 2018 · 11 comments
Labels
3.8 only security fixes type-bug An unexpected behavior, bug, or error

Comments

@piscvauyahoofr
Copy link
Mannequin

piscvauyahoofr mannequin commented Sep 7, 2018

BPO 34609
Nosy @brettcannon, @terryjreedy, @ncoghlan, @pitrou, @taleinat, @ericsnowcurrently, @pppery, @csabella, @tirkarthi
Superseder
  • bpo-33065: IDLE debugger: failure stepping through module loading
  • Files
  • test_main.py
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2018-09-23.18:14:03.033>
    created_at = <Date 2018-09-07.18:30:27.399>
    labels = ['type-bug', '3.8']
    title = 'Importing certain modules while debugging raises an exception'
    updated_at = <Date 2018-09-29.10:24:57.937>
    user = 'https://bugs.python.org/piscvauyahoofr'

    bugs.python.org fields:

    activity = <Date 2018-09-29.10:24:57.937>
    actor = 'pitrou'
    assignee = 'none'
    closed = True
    closed_date = <Date 2018-09-23.18:14:03.033>
    closer = 'terry.reedy'
    components = []
    creation = <Date 2018-09-07.18:30:27.399>
    creator = 'piscvau@yahoo.fr'
    dependencies = []
    files = ['47790']
    hgrepos = []
    issue_num = 34609
    keywords = []
    message_count = 10.0
    messages = ['324779', '324788', '324798', '324814', '324815', '324953', '326158', '326168', '326664', '326674']
    nosy_count = 10.0
    nosy_names = ['brett.cannon', 'terry.reedy', 'ncoghlan', 'pitrou', 'taleinat', 'eric.snow', 'ppperry', 'cheryl.sabella', 'xtreak', 'piscvau@yahoo.fr']
    pr_nums = []
    priority = 'normal'
    resolution = 'duplicate'
    stage = 'resolved'
    status = 'closed'
    superseder = '33065'
    type = 'behavior'
    url = 'https://bugs.python.org/issue34609'
    versions = ['Python 3.6', 'Python 3.8']

    @piscvauyahoofr
    Copy link
    Mannequin Author

    piscvauyahoofr mannequin commented Sep 7, 2018

    Idle is unable to import unittest. and raises an attributeError.
    Output of the Python shell Idle is included in the file bug idle.

    several files raise this exception
    on of them is included in this report
    the Trace for Idle is as follows :
    Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
    [GCC 7.3.0] on linux
    Type "copyright", "credits" or "license()" for more information.
    >>> 
    [DEBUG ON]
    >>> 
     RESTART: /home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py 
    Traceback (most recent call last):
      File "/home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py", line 5, in <module>
        import unittest
      File "<frozen importlib._bootstrap>", line 968, in _find_and_load
      File "<frozen importlib._bootstrap>", line 148, in __enter__
      File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
      File "<frozen importlib._bootstrap>", line 59, in __init__
      File "<frozen importlib._bootstrap>", line 59, in __init__
      File "/usr/lib/python3.6/bdb.py", line 51, in trace_dispatch
        return self.dispatch_line(frame)
      File "/usr/lib/python3.6/bdb.py", line 69, in dispatch_line
        self.user_line(frame)
      File "/usr/lib/python3.6/idlelib/debugger.py", line 24, in user_line
        self.gui.interaction(message, frame)
    AttributeError: '_ModuleLock' object has no attribute 'name'
    [DEBUG ON]

    Idle version is 3.6.5

    @piscvauyahoofr piscvauyahoofr mannequin added topic-IDLE type-bug An unexpected behavior, bug, or error labels Sep 7, 2018
    @terryjreedy
    Copy link
    Member

    I cannot confirm this. I tested 3.6.6 on Windows 10 with tem3.py containing

    import unittest
    print unittest

    I ran this twice, without and with debug on. The output:

    ======================== RESTART: F:\Python\a\tem3.py ========================

    <module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\__init__.py'>
    >>> 
    [DEBUG ON]
    >>> 
    ======================== RESTART: F:\Python\a\tem3.py 

    ========================
    <module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\init.py'>
    [DEBUG ON]

    >>

    On the second run, I hit the [Go] button on Debug Control. The minimal program above needs to be tested with 3.6.6 or 3.7.0 on Linux.

    @taleinat
    Copy link
    Contributor

    taleinat commented Sep 7, 2018

    Testing similarly to Terry, this doesn't reproduce with a recent build of master branch (3.8) on Ubuntu 16.04.

    I'll be able to try a build of older versions with IDLE only in a few days.

    @csabella
    Copy link
    Contributor

    csabella commented Sep 8, 2018

    I have Python 3.6.5 on Ubuntu 18.04. Initially, I was unable to recreate the issue. However, bpo-33065 from 4 months ago has the same trace (except on a different import), so I tried stepping through debug using the minimal example that Terry gave and successfully reproduced the trace.

    I then tried a new build of 3.8 on Ubuntu and was also able to reproduce by using 'step' while in debug.

    I also was able to recreate using 'step' under 3.6.3 on Windows 7.

    Note: At one point while stepping, the debug window shows a KeyError on unittest. You'll see it yourself, but it is interesting.

    @csabella
    Copy link
    Contributor

    csabella commented Sep 8, 2018

    Just to help track this down, 'import math' does not cause the trace, but 'import email' does. It seems to be modules that have a loader function. My guess (but I haven't tried it yet) is that this would happen if stepping through pdb outside of IDLE too.

    @csabella
    Copy link
    Contributor

    Following up, I was able to recreate under 3.8 using pdb in powershell, so it's not just in IDLE.

    Traceback (most recent call last):
      File "N:\projects\cpython\lib\pdb.py", line 1697, in main
        pdb._runscript(mainpyfile)
      File "N:\projects\cpython\lib\pdb.py", line 1566, in _runscript
        self.run(statement)
      File "N:\projects\cpython\lib\bdb.py", line 586, in run
        exec(cmd, globals, locals)
      File "<string>", line 1, in <module>
      File "c:\users\onedrive\desktop\step.py", line 1, in <module>
        import unittest
      File "<frozen importlib._bootstrap>", line 980, in _find_and_load
      File "<frozen importlib._bootstrap>", line 149, in __enter__
      File "<frozen importlib._bootstrap>", line 101, in acquire
    KeyError: 14104

    @taleinat taleinat added the 3.8 only security fixes label Sep 22, 2018
    @taleinat taleinat changed the title Idle Unitest Importing the certain modules while debugging raises an exception Sep 22, 2018
    @taleinat taleinat changed the title Importing the certain modules while debugging raises an exception Importing certain modules while debugging raises an exception Sep 22, 2018
    @terryjreedy terryjreedy removed their assignment Sep 23, 2018
    @pppery
    Copy link
    Mannequin

    pppery mannequin commented Sep 23, 2018

    The Pdb bug and the IDLE bug are unrelated to each other:

    Pdb fails because it is attempting to import the readline module every time its trace_dispatch is called, and the import implementation is not reentrant in that way.

    IDLE is crashing because the uninitialized _ModuleLock instance raised an error when repr is called, and the IDLE debugger lacks code to catch that exception.

    @terryjreedy
    Copy link
    Member

    The IDLE issue here is a duplicate of bpo-33065, where the OP said that only step, not go, created the problem. ppperry, thank you for the solution.

    Cheryl, please open a new issue for the pdb bug with a little more detail and ppperry's comment.

    @brettcannon
    Copy link
    Member

    The failure line that Cheryl found in her traceback is at https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 .

    Maybe Antoine has an idea since he did a bunch of work on import locking?

    @pitrou
    Copy link
    Member

    pitrou commented Sep 29, 2018

    I think pperry nailed it above:

    Pdb fails because it is attempting to import the readline module every time its trace_dispatch is called, and the import implementation is not reentrant in that way.

    More precisely, _ModuleLock.acquire() in https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 is not reentrant. If pdb steps into that function and tries to call it again by making an "import" call, _blocking_on[tid] will be overwritten and then deleted inside the nested call, so del _blocking_on in the enclosing call will raise a KeyError.

    I think the solution would be either one of:

    1. pdb avoids doing anything import-related as part of its step function
    2. pdb avoids stepping inside importlib internals (e.g. by blacklisting importlib modules)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @exarkun
    Copy link
    Contributor

    exarkun commented Jul 21, 2022

    Following up, I was able to recreate under 3.8 using pdb in powershell, so it's not just in IDLE.

    Traceback (most recent call last):
      File "N:\projects\cpython\lib\pdb.py", line 1697, in main
        pdb._runscript(mainpyfile)
      File "N:\projects\cpython\lib\pdb.py", line 1566, in _runscript
        self.run(statement)
      File "N:\projects\cpython\lib\bdb.py", line 586, in run
        exec(cmd, globals, locals)
      File "<string>", line 1, in <module>
      File "c:\users\onedrive\desktop\step.py", line 1, in <module>
        import unittest
      File "<frozen importlib._bootstrap>", line 980, in _find_and_load
      File "<frozen importlib._bootstrap>", line 149, in __enter__
      File "<frozen importlib._bootstrap>", line 101, in acquire
    KeyError: 14104
    

    This is the same as #91351

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.8 only security fixes type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants