New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"RuntimeError: Dictionary changed size during iteration" when copying a WeakValueDictionary #79796
Comments
I come across this issue recently when developing a multi-threaded PySide2 (Qt) application. When I'm calling .copy() on a WeakValueDictionary, there is a high chance that my application crashes with the following stack backtrace: ------ Traceback (most recent call last):
File "F:\angr\angr-management\angrmanagement\ui\widgets\qdisasm_graph.py", line 239, in mouseDoubleClickEvent
block.on_mouse_doubleclicked(event.button(), self._to_graph_pos(event.pos()))
File "F:\angr\angr-management\angrmanagement\ui\widgets\qblock.py", line 130, in on_mouse_doubleclicked
obj.on_mouse_doubleclicked(button, pos)
File "F:\angr\angr-management\angrmanagement\ui\widgets\qinstruction.py", line 128, in on_mouse_doubleclicked
op.on_mouse_doubleclicked(button, pos)
File "F:\angr\angr-management\angrmanagement\ui\widgets\qoperand.py", line 162, in on_mouse_doubleclicked
self.disasm_view.jump_to(self._branch_target, src_ins_addr=self.insn.addr)
File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", line 258, in jump_to
self._jump_to(addr)
File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", line 372, in _jump_to
self._display_function(function)
File "F:\angr\angr-management\angrmanagement\ui\views\disassembly_view.py", line 343, in _display_function
vr = self.workspace.instance.project.analyses.VariableRecoveryFast(the_func)
File "f:\angr\angr\angr\analyses\analysis.py", line 109, in __call__
oself.__init__(*args, **kwargs)
File "f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 618, in __init__
self._analyze()
File "f:\angr\angr\angr\analyses\forward_analysis.py", line 557, in _analyze
self._analysis_core_graph()
File "f:\angr\angr\angr\analyses\forward_analysis.py", line 580, in _analysis_core_graph
changed, output_state = self._run_on_node(n, job_state)
File "f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 712, in _run_on_node
input_state = prev_state.merge(input_state, successor=node.addr)
File "f:\angr\angr\angr\analyses\variable_recovery\variable_recovery_fast.py", line 488, in merge
merged_register_region = self.register_region.copy().replace(replacements).merge(other.register_region,
File "f:\angr\angr\angr\keyed_region.py", line 159, in copy
kr._object_mapping = self._object_mapping.copy()
File "D:\My Program Files\Python37\lib\weakref.py", line 174, in copy
for key, wr in self.data.items():
RuntimeError: dictionary changed size during iteration I went ahead and read the related methods in Lib\weakref.py, and it seems to me that the WeakValueDictionary.copy() method is missing the protection of an _IterationGuard: It is iterating through self.data.items(), which, might have entries removed because of GC during the iteration. It seems that this crash can be fixed by wrapping the iteration with `with _IterationGuard(self):`. It worked for me in my tests. If my above analysis is correct, the following methods all require protection of _IterationGuard (which are currently missing):
Please let me know if this is a legitimate issue, in which case I will be happy to provide a patch. Thanks. |
Hi, I think this is indeed a bug, I think the fix you suggested is the right solution. |
Hi! Seems like a bug. Please provide a patch. This can be apply to 3.8? |
Thanks for your reply. I'm preparing a PR. However, I'm not sure how to write a reliable test case to trigger the crash outside my application. I will submit the PR for now, and see if anyone on the mailing list has a better idea of what a reliable test case should look like. |
Just submitted a PR against the master branch on GitHub.
I think so. |
Just checked weakref.py on different branches, I think this bug exists in Python 2.7, 3.4, 3.5, and 3.6 as well. |
I think this issue needs to be reopened. The problem has not been solved completely. We experience a lot fewer crashes in weakref than before this fix, however, there are recursive situations in which copy() is invoked while iterating the WeakValueDictionary (e.g., in our case it is a signal/slot implementation where the slots are stored in a WeakValueDictionary). _commit_removals(), which is called at the beginning of the copy operation, might change the dictionary if there are items that are to be removed. If there is an ongoing iteration, the corresponding RuntimeError is raised. I haven't thought that through entirely, but I wonder whether the copy (and also deepcopy) operation could just blindly copy everything without "committing removals". After the copy, both instances would do their _commit_removals on their own upon access. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: