msg211737 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2014-02-20 16:47 |
Issuing the 'continue' pdb command with a lazy_import.py script as:
# START of lazy_import.py
import sys, pdb
for m in sys.modules:
if m == 'sys':
pdb.set_trace()
# END of lazy_import.py
gives the following output:
$ python lazy_import.py
> lazy_import.py(3)<module>()
-> for m in sys.modules:
(Pdb) continue
Traceback (most recent call last):
File "lazy_import.py", line 3, in <module>
for m in sys.modules:
RuntimeError: dictionary changed size during iteration
|
msg236617 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2015-02-25 20:30 |
I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.
|
msg236634 - (view) |
Author: Josh Rosenberg (josh.r) * |
Date: 2015-02-26 00:14 |
Is this worth fixing? Delve into the innards and you get weird behaviors. If you really need to iterate sys.modules (the example clearly doesn't, but that doesn't mean no one would), you could just copy the parts you need beforehand to get a consistent view, e.g.
for name, mod in tuple(sys.modules.items()):
Lazy module imports aren't always a bad thing if the module in question isn't needed for the common functionality, only specific subsets that may never be used.
|
msg236662 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2015-02-26 11:03 |
> I can reproduce this on Windows 8.1 with 3.4.3 but cannot do so with 3.5.0a1.
I can still reproduce this on linux on today's tip:
'3.5.0a1+ (default:7185a35fb293, Feb 26 2015, 11:27:11) \n[GCC 4.9.2 20150204 (prerelease)]'.
Maybe you tried reproducing it with the interactive interpreter. When this code
is run from the interactive interpreter and not as a script as suggested in the
initial post, then readline is already imported by the interpreter at startup
and the RuntimeError is not raised.
Also interestingly, the bdb module takes lot of steps to ensure that linecache
and reprlib are lazyly imported, but pdb and bdb cannot do anything interesting
without these modules.
|
msg236674 - (view) |
Author: Mark Lawrence (BreamoreBoy) * |
Date: 2015-02-26 15:21 |
My tests were done with a script from the command line, not an interactive prompt.
|
msg236677 - (view) |
Author: Eric Snow (eric.snow) * |
Date: 2015-02-26 15:31 |
I haven't looked to closely but I'm guessing that pdb.set_trace() causes something to get imported (i.e. there's an import statement in a function body somewhere). Consequently sys.modules is updated (by that "distant" import statement) while you are iterating over it here.
In that case the behavior you are seeing is correct, even if not obvious or even desirable. It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).
|
msg236735 - (view) |
Author: Xavier de Gaye (xdegaye) * |
Date: 2015-02-27 09:39 |
> In that case the behavior you are seeing is correct, even if not obvious or even desirable. It will happen any time you are looping over sys.modules and call a function/method which has a function-scoped import statement for a module that hasn't been imported yet (or calls another function that does so, etc.).
I do not agree. Suppose there is some piece of code that loops over sys.modules without doing any lazy import (and thus without raising the RuntimeError "dictionary changed size during iteration"). Now, if you insert a pdb.set_trace() in that code, you will get the RuntimeError and this is not correct.
|
msg296873 - (view) |
Author: Louie Lu (louielu) * |
Date: 2017-06-26 10:59 |
The lazy import cause by two modules, readline and _bootlocale.
readline: in __init__
try:
import readline
readline.set_completer_delims(' \t\n`@#$%^&*()=+[{]}\\|;:\'",<>?')
_bootlocale: in __init__
with open(os.path.join(envHome, '.pdbrc')) as rcFile:
with open('.pdbrc') as rcFile
Easiest way to eliminated it to move import readline and import _bootlocale to the top of the file.
|
msg398667 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2021-07-31 22:06 |
I agree with both Eric and Xavier - any library calling a function that modifies sys.modules will see this issue, but pdb should try not to modify program semantics like this.
It's fixed if we move the readline import as Louie suggested (the _bootlocale import was probably removed by now, I don't see it).
|
msg398727 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2021-08-02 06:11 |
Having a side effect at import time is not good. It will interfere with programs which just import pdb, but not use it.
There are two other options:
1. Import readline at top level, but call set_completer_delims() lazily.
2. Do not import readline at all. Call set_completer_delims() only if readline is already imported (sys.modules.get('readline') is not None).
I prefer the latter one.
|
msg398768 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2021-08-02 13:46 |
I think option 2 changes the current behaviour, because cmd.Cmd.cmdloop will import readline later, and the set_completer_delims() call would never happen even though readline is used. I'll update the patch to do option 1.
|
msg398832 - (view) |
Author: Serhiy Storchaka (serhiy.storchaka) * |
Date: 2021-08-03 15:15 |
My concern about importing readline at the top of the module is that importing readline has a side effect. The only module which imports it unconditionally at the top is rlcompleter. In all other places it is imported lazily.
And importing readline ahead may not fix the original issue, because there are other modules lazily imported in pdb: runpy, shlex, pydoc.
Perhaps we should just say that sys.modules should not be iterated directly, you should first make a copy. There are other issues about RuntimeError raised during iterating sys.modules.
|
msg398833 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2021-08-03 15:18 |
Can we make sys.modules automatically return an iterator that has a copy of its contents? Otherwise it's an odd API - it's iterable but we tell people not to iterate it.
|
msg398852 - (view) |
Author: Irit Katriel (iritkatriel) * |
Date: 2021-08-03 22:26 |
Closing as this is not a problem with pdb, it's a general problem with iterating sys.modules while doing things that can cause imports.
|
|
Date |
User |
Action |
Args |
2022-04-11 14:57:58 | admin | set | github: 64902 |
2021-08-03 22:26:46 | iritkatriel | set | status: open -> closed resolution: wont fix messages:
+ msg398852
stage: patch review -> resolved |
2021-08-03 15:18:21 | iritkatriel | set | messages:
+ msg398833 |
2021-08-03 15:15:28 | serhiy.storchaka | set | messages:
+ msg398832 |
2021-08-02 13:46:08 | iritkatriel | set | messages:
+ msg398768 |
2021-08-02 06:11:15 | serhiy.storchaka | set | nosy:
+ serhiy.storchaka messages:
+ msg398727
|
2021-07-31 22:06:46 | iritkatriel | set | messages:
+ msg398667 versions:
+ Python 3.9, Python 3.10, Python 3.11, - Python 3.6, Python 3.7 |
2021-07-31 21:55:14 | iritkatriel | set | keywords:
+ patch nosy:
+ iritkatriel
pull_requests:
+ pull_request26035 stage: patch review |
2017-06-26 20:12:43 | BreamoreBoy | set | nosy:
- BreamoreBoy
|
2017-06-26 10:59:18 | louielu | set | versions:
+ Python 3.6, Python 3.7, - Python 3.4, Python 3.5 |
2017-06-26 10:59:07 | louielu | set | nosy:
+ louielu messages:
+ msg296873
|
2015-02-27 09:39:51 | xdegaye | set | messages:
+ msg236735 |
2015-02-26 15:31:07 | eric.snow | set | nosy:
+ eric.snow messages:
+ msg236677
|
2015-02-26 15:21:08 | BreamoreBoy | set | messages:
+ msg236674 |
2015-02-26 11:03:06 | xdegaye | set | messages:
+ msg236662 |
2015-02-26 00:14:43 | josh.r | set | nosy:
+ josh.r messages:
+ msg236634
|
2015-02-25 20:30:26 | BreamoreBoy | set | nosy:
+ BreamoreBoy
messages:
+ msg236617 versions:
+ Python 3.5 |
2014-02-20 16:47:20 | xdegaye | create | |