New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
inspect.getmodule fails when module imports change sys.modules #57696
Comments
The inspect.getmodule function crashes if packages are installed that futz with sys.modules while they are being tested for module status or the like. I'm not actually sure which packages are doing this, but the symptom is the for loop over sys.modules raises an Exception because it is modified while the loop is running. This is *not* a problem in Python 2.x because sys.modules.items() returns a copy of the dictionary instead of an iterator, and 3.x changes that behavior. The comment above the for loop makes it clear that the expected behavior is a copy rather than an iterator, so the attached patch corrects the problem by simply wrapping the items() call in list(). |
You are certainly right, but I wonder how this can happen. |
Maybe it can be caused by an installation happening during the loop. I agree with Erik’s reading of the comment and patch, and don’t think a test is needed. |
The package that triggers it for me is the py (http://pypi.python.org/pypi/py) package - when in gets imported, it does some trick with sys.modules that is in place to get around some pickling restriction, but that means sys.modules is altered during the import of the |
When a package is imported sys.modules changes... nothing special here. Patch looks good to me as well. |
New changeset 2ef359d7a2e9 by Éric Araujo in branch '3.2': |
Committed, thanks. |
list(sys.modules.items()) still raises RuntimeError: dictionary changed size during iteration when another thread imports a module. I would assume dict.copy() is thread-safe so a working fix could use sys.modules.copy().items() I hit this bug when printing the name of the caller function using |
This bug seems to be still there. I had an application fail with this same error in inspect.getouterframes with Python 3.6.2. As far as I could trace it, during iteration over sys.modules _tracemalloc and tracemalloc were added, not quite sure from where (maybe the warnings module ?) |
Could you give code to reproducer the problem, if possible without third-party dependencies? |
dict.copy() is not thread-safe still (but it can be made thread-safe). list(dict) is thread-safe. It copies a list of keys only. |
I cannot reproduce. In fact I cannot even get list(d.items()) |
fyi - we just had a test run into this (in a flaky manner - definitely a race condition) at work:
We haven't diagnosed what was leading to it though. Trust in the ability to use inspect.stack() -> ... -> inspect.getmodule() in multithreaded code is on the way out as a workaround. (this was on 3.6.7) A workaround we could checkin without consequences should be to change list(sys.modules.items()) into list(sys.modules.copy().items()). I was a bit surprised to see this happen at all, list(dict.items()) seems like it should've been done entirely in C with the GIL held the entire time. but maybe I'm just missing where the GIL would be released in the calls to exhause the iterator made by https://github.com/python/cpython/blob/master/Objects/listobject.c ? |
Serhiy: Why is dict.copy() not thread safe? if what you say of list(dict) being safe, iterating over that and looking up the keys would work. But all of this is entirely non-obvious to the reader of the code. all of these _look_ like they should be safe. We should make dict.copy() safe and document the guarantee as such as that one could at least be explained when used for that purpose. |
Testing by changing list(sys.modules.items()) to sys.modules.copy().items() internally with a large integration test that was reliably flaky on this line before shows that the .copy().items() worked. The test is reliable again. So I've gone ahead and pushed those changes in. PyDict_Copy()'s implementation at first ~5 minute glance did not appear to have calls to code I'd expect to re-enter Python releasing the GIL. But I didn't try to do a deep dive. It works for us and is logically equivalent. |
If anyone else has a way to reproduce this issue, please run your tests with a 3.7/3.8/3.9 interpreter with the fix I committed applied and report back if you still see this failure on that line. I believe this to be fixed based on my own testing so I am closing it out. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: