classification
Title: RuntimeError caused by lazy imports in pdb
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: eric.snow, georg.brandl, iritkatriel, josh.r, louielu, serhiy.storchaka, xdegaye
Priority: normal Keywords: patch

Created on 2014-02-20 16:47 by xdegaye, last changed 2021-08-03 22:26 by iritkatriel. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27520 closed iritkatriel, 2021-07-31 21:55
Messages (14)
msg211737 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2014-02-20 16:47
Issuing the 'continue' pdb command with a lazy_import.py script as:

# START of lazy_import.py
import sys, pdb

for m in sys.modules:
    if m == 'sys':
        pdb.set_trace()

# END of lazy_import.py


gives the following output:

$ python lazy_import.py
> lazy_import.py(3)<module>()
-> for m in sys.modules:
(Pdb) continue
Traceback (most recent call last):
  File "lazy_import.py", line 3, in <module>
    for m in sys.modules:
RuntimeError: dictionary changed size during iteration
msg236617 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-25 20:30
I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.
msg236634 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2015-02-26 00:14
Is this worth fixing? Delve into the innards and you get weird behaviors. If you really need to iterate sys.modules (the example clearly doesn't, but that doesn't mean no one would), you could just copy the parts you need beforehand to get a consistent view, e.g.

for name, mod in tuple(sys.modules.items()):

Lazy module imports aren't always a bad thing if the module in question isn't needed for the common functionality, only specific subsets that may never be used.
msg236662 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-02-26 11:03
> I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.

I can still reproduce this on linux on today's tip:
'3.5.0a1+ (default:7185a35fb293, Feb 26 2015, 11:27:11) \n[GCC 4.9.2 20150204 (prerelease)]'.

Maybe you tried reproducing it with the interactive interpreter. When this code
is run from the interactive interpreter and not as a script as suggested in the
initial post, then readline is already imported by the interpreter at startup
and the RuntimeError is not raised.

Also interestingly, the bdb module takes lot of steps to ensure that linecache
and reprlib are lazyly imported, but pdb and bdb cannot do anything interesting
without these modules.
msg236674 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2015-02-26 15:21
My tests were done with a script from the command line, not an interactive prompt.
msg236677 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2015-02-26 15:31
I haven't looked to closely but I'm guessing that pdb.set_trace() causes something to get imported (i.e. there's an import statement in a function body somewhere).  Consequently sys.modules is updated (by that "distant" import statement) while you are iterating over it here.

In that case the behavior you are seeing is correct, even if not obvious or even desirable.  It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).
msg236735 - (view) Author: Xavier de Gaye (xdegaye) * (Python triager) Date: 2015-02-27 09:39
> In that case the behavior you are seeing is correct, even if not obvious or even desirable.  It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).

I do not agree. Suppose there is some piece of code that loops over sys.modules without doing any lazy import (and thus without raising the RuntimeError "dictionary changed size during iteration"). Now, if you insert a pdb.set_trace() in that code, you will get the RuntimeError and this is not correct.
msg296873 - (view) Author: Louie Lu (louielu) * Date: 2017-06-26 10:59
The lazy import cause by two modules, readline and _bootlocale.

readline: in __init__

   try:
       import readline
       readline.set_completer_delims(' \t\n`@#$%^&*()=+[{]}\\|;:\'",<>?')

_bootlocale: in __init__

    with open(os.path.join(envHome, '.pdbrc')) as rcFile:
    with open('.pdbrc') as rcFile

Easiest way to eliminated it to move import readline and import _bootlocale to the top of the file.
msg398667 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-07-31 22:06
I agree with both Eric and Xavier - any library calling a function that modifies sys.modules will see this issue, but pdb should try not to modify program semantics like this. 

It's fixed if we move the readline import as Louie suggested (the _bootlocale import was probably removed by now, I don't see it).
msg398727 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-02 06:11
Having a side effect at import time is not good. It will interfere with programs which just import pdb, but not use it.

There are two other options:

1. Import readline at top level, but call set_completer_delims() lazily.

2. Do not import readline at all. Call set_completer_delims() only if readline is already imported (sys.modules.get('readline') is not None).

I prefer the latter one.
msg398768 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-08-02 13:46
I think option 2 changes the current behaviour, because cmd.Cmd.cmdloop will import readline later, and the set_completer_delims() call would never happen even though readline is used. I'll update the patch to do option 1.
msg398832 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-03 15:15
My concern about importing readline at the top of the module is that importing readline has a side effect. The only module which imports it unconditionally at the top is rlcompleter. In all other places it is imported lazily.

And importing readline ahead may not fix the original issue, because there are other modules lazily imported in pdb: runpy, shlex, pydoc.

Perhaps we should just say that sys.modules should not be iterated directly, you should first make a copy. There are other issues about RuntimeError raised during iterating sys.modules.
msg398833 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-08-03 15:18
Can we make sys.modules automatically return an iterator that has a copy of its contents?  Otherwise it's an odd API - it's iterable but we tell people not to iterate it.
msg398852 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-08-03 22:26
Closing as this is not a problem with pdb, it's a general problem with iterating sys.modules while doing things that can cause imports.
History
Date User Action Args
2021-08-03 22:26:46iritkatrielsetstatus: open -> closed
resolution: wont fix
messages: + msg398852

stage: patch review -> resolved
2021-08-03 15:18:21iritkatrielsetmessages: + msg398833
2021-08-03 15:15:28serhiy.storchakasetmessages: + msg398832
2021-08-02 13:46:08iritkatrielsetmessages: + msg398768
2021-08-02 06:11:15serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg398727
2021-07-31 22:06:46iritkatrielsetmessages: + msg398667
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 3.6, Python 3.7
2021-07-31 21:55:14iritkatrielsetkeywords: + patch
nosy: + iritkatriel

pull_requests: + pull_request26035
stage: patch review
2017-06-26 20:12:43BreamoreBoysetnosy: - BreamoreBoy
2017-06-26 10:59:18louielusetversions: + Python 3.6, Python 3.7, - Python 3.4, Python 3.5
2017-06-26 10:59:07louielusetnosy: + louielu
messages: + msg296873
2015-02-27 09:39:51xdegayesetmessages: + msg236735
2015-02-26 15:31:07eric.snowsetnosy: + eric.snow
messages: + msg236677
2015-02-26 15:21:08BreamoreBoysetmessages: + msg236674
2015-02-26 11:03:06xdegayesetmessages: + msg236662
2015-02-26 00:14:43josh.rsetnosy: + josh.r
messages: + msg236634
2015-02-25 20:30:26BreamoreBoysetnosy: + BreamoreBoy

messages: + msg236617
versions: + Python 3.5
2014-02-20 16:47:20xdegayecreate