This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: RuntimeError in pickle.whichmodule when sys.modules if mutated
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.5
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Olivier.Grisel, alexandre.vassalotti, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2014-07-02 13:03 by Olivier.Grisel, last changed 2022-04-11 14:58 by admin. This issue is now closed.

File name Uploaded Description Edit
pickle_whichmodule.patch Olivier.Grisel, 2014-07-02 13:03 review
pickle_whichmodule_20140703.patch Olivier.Grisel, 2014-07-03 15:04 review
Messages (6)
msg222099 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-07-02 13:03
`pickle.whichmodule` performs an iteration over `sys.modules` and tries to perform `getattr` calls on those modules. Unfortunately some modules such as those from the `six.moves` dynamic module can trigger imports when calling `getattr` on them, hence mutating the `sys.modules` dict and causing a `RuntimeError: dictionary changed size during iteration`.

This would also render `pickle.whichmodule` more thread-safe and cause concurrent thread perform new module imports and `whichmodule` calls.

The attach patch protect the iterator by copying the dict items into a fixed list.

I could write a tests involving dynamic module definitions as done in `six.moves` but it sounds very complicated for such a trivial fix.
msg222188 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-03 14:39
Sounds fine, but you should at least add a comment explaining the list() call.
msg222193 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-07-03 15:04
New version of the patch to add an inline comment.
msg228488 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-10-04 20:17
New changeset 86ba3bdfac15 by Antoine Pitrou in branch '3.4':
Issue #21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.

New changeset d748a3503ad5 by Antoine Pitrou in branch 'default':
Issue #21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.
msg228489 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-04 20:18
Sorry for the delay, Olivier. Your patch is now pushed. Thank you!
msg228646 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-10-06 07:25
No problem. Thanks Antoine for the review!
Date User Action Args
2022-04-11 14:58:05adminsetgithub: 66104
2014-10-06 15:10:31vstinnerlinkissue22292 superseder
2014-10-06 07:25:56Olivier.Griselsetmessages: + msg228646
2014-10-04 20:18:24pitrousetstatus: open -> closed
resolution: fixed
messages: + msg228489

stage: patch review -> resolved
2014-10-04 20:17:51python-devsetnosy: + python-dev
messages: + msg228488
2014-07-03 15:04:30Olivier.Griselsetfiles: + pickle_whichmodule_20140703.patch

messages: + msg222193
2014-07-03 14:39:30pitrousetnosy: + pitrou, alexandre.vassalotti
messages: + msg222188

type: crash -> behavior
stage: patch review
2014-07-02 13:03:12Olivier.Griselcreate