classification
Title: RuntimeError in pickle.whichmodule when sys.modules if mutated
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Olivier.Grisel, alexandre.vassalotti, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2014-07-02 13:03 by Olivier.Grisel, last changed 2014-10-06 07:25 by Olivier.Grisel. This issue is now closed.

Files
File name Uploaded Description Edit
pickle_whichmodule.patch Olivier.Grisel, 2014-07-02 13:03 review
pickle_whichmodule_20140703.patch Olivier.Grisel, 2014-07-03 15:04 review
Messages (6)
msg222099 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-07-02 13:03
`pickle.whichmodule` performs an iteration over `sys.modules` and tries to perform `getattr` calls on those modules. Unfortunately some modules such as those from the `six.moves` dynamic module can trigger imports when calling `getattr` on them, hence mutating the `sys.modules` dict and causing a `RuntimeError: dictionary changed size during iteration`.

This would also render `pickle.whichmodule` more thread-safe and cause concurrent thread perform new module imports and `whichmodule` calls.

The attach patch protect the iterator by copying the dict items into a fixed list.

I could write a tests involving dynamic module definitions as done in `six.moves` but it sounds very complicated for such a trivial fix.
msg222188 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-03 14:39
Sounds fine, but you should at least add a comment explaining the list() call.
msg222193 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-07-03 15:04
New version of the patch to add an inline comment.
msg228488 - (view) Author: Roundup Robot (python-dev) Date: 2014-10-04 20:17
New changeset 86ba3bdfac15 by Antoine Pitrou in branch '3.4':
Issue #21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.
https://hg.python.org/cpython/rev/86ba3bdfac15

New changeset d748a3503ad5 by Antoine Pitrou in branch 'default':
Issue #21905: Avoid RuntimeError in pickle.whichmodule() when sys.modules is mutated while iterating.
https://hg.python.org/cpython/rev/d748a3503ad5
msg228489 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-04 20:18
Sorry for the delay, Olivier. Your patch is now pushed. Thank you!
msg228646 - (view) Author: Olivier Grisel (Olivier.Grisel) * Date: 2014-10-06 07:25
No problem. Thanks Antoine for the review!
History
Date User Action Args
2014-10-06 15:10:31haypolinkissue22292 superseder
2014-10-06 07:25:56Olivier.Griselsetmessages: + msg228646
2014-10-04 20:18:24pitrousetstatus: open -> closed
resolution: fixed
messages: + msg228489

stage: patch review -> resolved
2014-10-04 20:17:51python-devsetnosy: + python-dev
messages: + msg228488
2014-07-03 15:04:30Olivier.Griselsetfiles: + pickle_whichmodule_20140703.patch

messages: + msg222193
2014-07-03 14:39:30pitrousetnosy: + pitrou, alexandre.vassalotti
messages: + msg222188

type: crash -> behavior
stage: patch review
2014-07-02 13:03:12Olivier.Griselcreate