classification
Title: RuntimeError caused by lazy imports in pdb
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, georg.brandl, josh.r, louielu, xdegaye
Priority: normal Keywords:

Created on 2014-02-20 16:47 by xdegaye, last changed 2017-06-26 20:12 by BreamoreBoy.

Messages (8)
msg211737 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-02-20 16:47
Issuing the 'continue' pdb command with a lazy_import.py script as:

# START of lazy_import.py
import sys, pdb

for m in sys.modules:
    if m == 'sys':
        pdb.set_trace()

# END of lazy_import.py


gives the following output:

$ python lazy_import.py
> lazy_import.py(3)<module>()
-> for m in sys.modules:
(Pdb) continue
Traceback (most recent call last):
  File "lazy_import.py", line 3, in <module>
    for m in sys.modules:
RuntimeError: dictionary changed size during iteration
msg236617 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-25 20:30
I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.
msg236634 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2015-02-26 00:14
Is this worth fixing? Delve into the innards and you get weird behaviors. If you really need to iterate sys.modules (the example clearly doesn't, but that doesn't mean no one would), you could just copy the parts you need beforehand to get a consistent view, e.g.

for name, mod in tuple(sys.modules.items()):

Lazy module imports aren't always a bad thing if the module in question isn't needed for the common functionality, only specific subsets that may never be used.
msg236662 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-02-26 11:03
> I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.

I can still reproduce this on linux on today's tip:
'3.5.0a1+ (default:7185a35fb293, Feb 26 2015, 11:27:11) \n[GCC 4.9.2 20150204 (prerelease)]'.

Maybe you tried reproducing it with the interactive interpreter. When this code
is run from the interactive interpreter and not as a script as suggested in the
initial post, then readline is already imported by the interpreter at startup
and the RuntimeError is not raised.

Also interestingly, the bdb module takes lot of steps to ensure that linecache
and reprlib are lazyly imported, but pdb and bdb cannot do anything interesting
without these modules.
msg236674 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-26 15:21
My tests were done with a script from the command line, not an interactive prompt.
msg236677 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-02-26 15:31
I haven't looked to closely but I'm guessing that pdb.set_trace() causes something to get imported (i.e. there's an import statement in a function body somewhere).  Consequently sys.modules is updated (by that "distant" import statement) while you are iterating over it here.

In that case the behavior you are seeing is correct, even if not obvious or even desirable.  It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).
msg236735 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-02-27 09:39
> In that case the behavior you are seeing is correct, even if not obvious or even desirable.  It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).

I do not agree. Suppose there is some piece of code that loops over sys.modules without doing any lazy import (and thus without raising the RuntimeError "dictionary changed size during iteration"). Now, if you insert a pdb.set_trace() in that code, you will get the RuntimeError and this is not correct.
msg296873 - (view) Author: Louie Lu (louielu) * Date: 2017-06-26 10:59
The lazy import cause by two modules, readline and _bootlocale.

readline: in __init__

   try:
       import readline
       readline.set_completer_delims(' \t\n`@#$%^&*()=+[{]}\\|;:\'",<>?')

_bootlocale: in __init__

    with open(os.path.join(envHome, '.pdbrc')) as rcFile:
    with open('.pdbrc') as rcFile

Easiest way to eliminated it to move import readline and import _bootlocale to the top of the file.
History
Date User Action Args
2017-06-26 20:12:43BreamoreBoysetnosy: - BreamoreBoy
2017-06-26 10:59:18louielusetversions: + Python 3.6, Python 3.7, - Python 3.4, Python 3.5
2017-06-26 10:59:07louielusetnosy: + louielu
messages: + msg296873
2015-02-27 09:39:51xdegayesetmessages: + msg236735
2015-02-26 15:31:07eric.snowsetnosy: + eric.snow
messages: + msg236677
2015-02-26 15:21:08BreamoreBoysetmessages: + msg236674
2015-02-26 11:03:06xdegayesetmessages: + msg236662
2015-02-26 00:14:43josh.rsetnosy: + josh.r
messages: + msg236634
2015-02-25 20:30:26BreamoreBoysetnosy: + BreamoreBoy

messages: + msg236617
versions: + Python 3.5
2014-02-20 16:47:20xdegayecreate