classification
Title: Importing certain modules while debugging raises an exception
Type: behavior Stage: resolved
Components: Versions: Python 3.8, Python 3.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: IDLE debugger: failure stepping through module loading
View: 33065
Assigned To: Nosy List: brett.cannon, cheryl.sabella, eric.snow, ncoghlan, piscvau@yahoo.fr, pitrou, ppperry, taleinat, terry.reedy, xtreak
Priority: normal Keywords:

Created on 2018-09-07 18:30 by piscvau@yahoo.fr, last changed 2018-09-29 10:24 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
test_main.py piscvau@yahoo.fr, 2018-09-07 18:30
Messages (10)
msg324779 - (view) Author: (piscvau@yahoo.fr) Date: 2018-09-07 18:30
Idle is unable to import unittest. and raises an attributeError.
Output of the Python shell Idle is included in the file bug idle.

several files raise this exception
on of them is included in this report
the Trace for Idle is as follows :
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0] on linux
Type "copyright", "credits" or "license()" for more information.
>>> 
[DEBUG ON]
>>> 
 RESTART: /home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py 
Traceback (most recent call last):
  File "/home/odile/Documents/dev_logiciels/pyhton/openclassrooms/labyrinthe/roboc2/test/test_carte.py", line 5, in <module>
    import unittest
  File "<frozen importlib._bootstrap>", line 968, in _find_and_load
  File "<frozen importlib._bootstrap>", line 148, in __enter__
  File "<frozen importlib._bootstrap>", line 174, in _get_module_lock
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "<frozen importlib._bootstrap>", line 59, in __init__
  File "/usr/lib/python3.6/bdb.py", line 51, in trace_dispatch
    return self.dispatch_line(frame)
  File "/usr/lib/python3.6/bdb.py", line 69, in dispatch_line
    self.user_line(frame)
  File "/usr/lib/python3.6/idlelib/debugger.py", line 24, in user_line
    self.gui.interaction(message, frame)
AttributeError: '_ModuleLock' object has no attribute 'name'
[DEBUG ON]

Idle version is 3.6.5
msg324788 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-09-07 19:41
I cannot confirm this.  I tested 3.6.6 on Windows 10 with tem3.py containing

import unittest
print unittest

I ran this twice, without and with debug on.  The output:

======================== RESTART: F:\Python\a\tem3.py ========================
<module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\__init__.py'>
>>> 
[DEBUG ON]
>>> 
======================== RESTART: F:\Python\a\tem3.py ========================
<module 'unittest' from 'C:\\Programs\\Python36\\lib\\unittest\\__init__.py'>
[DEBUG ON]
>>>

On the second run, I hit the [Go] button on Debug Control.  The minimal program above needs to be tested with 3.6.6 or 3.7.0 on Linux.
msg324798 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2018-09-07 21:38
Testing similarly to Terry, this doesn't reproduce with a recent build of master branch (3.8) on Ubuntu 16.04.

I'll be able to try a build of older versions with IDLE only in a few days.
msg324814 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-09-08 00:42
I have Python 3.6.5 on Ubuntu 18.04.  Initially, I was unable to recreate the issue.  However, issue33065 from 4 months ago has the same trace (except on a different import), so I tried stepping through debug using the minimal example that Terry gave and successfully reproduced the trace.

I then tried a new build of 3.8 on Ubuntu and was also able to reproduce by using 'step' while in debug.

I also was able to recreate using 'step' under 3.6.3 on Windows 7.

Note: At one point while stepping, the debug window shows a KeyError on unittest.  You'll see it yourself, but it is interesting.
msg324815 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-09-08 00:52
Just to help track this down, 'import math' does not cause the trace, but 'import email' does.  It seems to be modules that have a loader function.  My guess (but I haven't tried it yet) is that this would happen if stepping through pdb outside of IDLE too.
msg324953 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2018-09-10 21:10
Following up, I was able to recreate under 3.8 using pdb in powershell, so it's not just in IDLE.

Traceback (most recent call last):
  File "N:\projects\cpython\lib\pdb.py", line 1697, in main
    pdb._runscript(mainpyfile)
  File "N:\projects\cpython\lib\pdb.py", line 1566, in _runscript
    self.run(statement)
  File "N:\projects\cpython\lib\bdb.py", line 586, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\users\onedrive\desktop\step.py", line 1, in <module>
    import unittest
  File "<frozen importlib._bootstrap>", line 980, in _find_and_load
  File "<frozen importlib._bootstrap>", line 149, in __enter__
  File "<frozen importlib._bootstrap>", line 101, in acquire
KeyError: 14104
msg326158 - (view) Author: (ppperry) Date: 2018-09-23 15:13
The Pdb bug and the IDLE bug are unrelated to each other:

   Pdb fails because it is attempting to import the readline module every time its `trace_dispatch` is called, and the import implementation is not reentrant in that way.

   IDLE is crashing because the uninitialized _ModuleLock instance raised an error when `repr` is called, and the IDLE debugger lacks code to catch that exception.
msg326168 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2018-09-23 18:14
The IDLE issue here is a duplicate of #33065, where the OP said that only step, not go, created the problem.  ppperry, thank you for the solution.

Cheryl, please open a new issue for the pdb bug with a little more detail and ppperry's comment.
msg326664 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-09-29 00:28
The failure line that Cheryl found in her traceback is at https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 .

Maybe Antoine has an idea since he did a bunch of work on import locking?
msg326674 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-09-29 10:24
I think pperry nailed it above:

> Pdb fails because it is attempting to import the readline module every time its `trace_dispatch` is called, and the import implementation is not reentrant in that way.

More precisely, _ModuleLock.acquire() in https://github.com/python/cpython/blob/master/Lib/importlib/_bootstrap.py#L101 is not reentrant.  If pdb steps into that function and tries to call it again by making an "import" call, `_blocking_on[tid]` will be overwritten and then deleted inside the nested call, so `del _blocking_on` in the enclosing call will raise a KeyError.

I think the solution would be either one of:
1) pdb avoids doing anything import-related as part of its step function
2) pdb avoids stepping inside importlib internals (e.g. by blacklisting importlib modules)
History
Date User Action Args
2018-09-29 10:24:57pitrousetmessages: + msg326674
2018-09-29 00:28:19brett.cannonsetnosy: + pitrou
messages: + msg326664
2018-09-23 18:14:03terry.reedysetstatus: open -> closed
superseder: IDLE debugger: failure stepping through module loading
messages: + msg326168

resolution: duplicate
stage: resolved
2018-09-23 15:13:50ppperrysetnosy: + ppperry
messages: + msg326158
2018-09-23 06:32:17terry.reedysetassignee: terry.reedy ->
components: - IDLE
2018-09-23 05:35:59serhiy.storchakasetnosy: + brett.cannon, ncoghlan, eric.snow
2018-09-23 04:48:58xtreaksetnosy: + xtreak
2018-09-22 17:56:08taleinatsettitle: Importing the certain modules while debugging raises an exception -> Importing certain modules while debugging raises an exception
2018-09-22 17:55:42taleinatsettitle: Idle Unitest -> Importing the certain modules while debugging raises an exception
2018-09-22 17:31:11taleinatsetversions: + Python 3.8
2018-09-10 21:10:22cheryl.sabellasetmessages: + msg324953
2018-09-08 00:52:46cheryl.sabellasetmessages: + msg324815
2018-09-08 00:42:30cheryl.sabellasetmessages: + msg324814
2018-09-07 21:38:02taleinatsetmessages: + msg324798
2018-09-07 19:41:50terry.reedysetnosy: + taleinat, cheryl.sabella
messages: + msg324788
2018-09-07 18:30:27piscvau@yahoo.frcreate