classification
Title: pickle whichmodule RuntimeError
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: duplicate
Dependencies: Superseder: RuntimeError in pickle.whichmodule when sys.modules if mutated
View: 21905
Assigned To: Nosy List: attilio.dinisio, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2014-08-27 21:51 by attilio.dinisio, last changed 2014-10-06 15:10 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
pickle.py.diff attilio.dinisio, 2014-08-28 20:01 review
Messages (5)
msg226005 - (view) Author: Attilio Di Nisio (attilio.dinisio) * Date: 2014-08-27 21:51
= Error message =
pickle.dumps and pickle.dump generate the following error:

  File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
    for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration


= Test case = 
The test case is as follows:

    import pickle
    import sys
    import numpy #version 1.8.1
    import scipy.signal #version 0.13.3
    pickle.dumps(numpy.cos)

Sometimes this results in the error

Traceback (most recent call last):
  File "whichmodule_bug.py", line 13, in <module>
    pickle.dumps(numpy.cos)
  File "/usr/lib/python3/dist-packages/numpy/core/__init__.py", line 61, in _ufunc_reduce
    return _ufunc_reconstruct, (whichmodule(func, name), name)
  File "/usr/lib/python3.4/pickle.py", line 283, in whichmodule
    for module_name, module in sys.modules.items():
RuntimeError: dictionary changed size during iteration

NOTE: This is not necessarily a problem due to numpy and scipy packages. Explanation follows.


= Relevant code = 
The last version of pickle http://hg.python.org/cpython/file/f6f691ff27b9/Lib/pickle.py contains:
...

def _getattribute(obj, name, allow_qualname=False):
    dotted_path = name.split(".")
    if not allow_qualname and len(dotted_path) > 1:
        raise AttributeError("Can't get qualified attribute {!r} on {!r}; " +
                             "use protocols >= 4 to enable support"
                             .format(name, obj))
    for subpath in dotted_path:
        if subpath == '<locals>':
            raise AttributeError("Can't get local attribute {!r} on {!r}"
                                 .format(name, obj))
        try:
            obj = getattr(obj, subpath)
        except AttributeError:
            raise AttributeError("Can't get attribute {!r} on {!r}"
                                 .format(name, obj))
    return obj

def whichmodule(obj, name, allow_qualname=False):
    """Find the module an object belong to."""
    module_name = getattr(obj, '__module__', None)
    if module_name is not None:
        return module_name
    for module_name, module in sys.modules.items():
        if module_name == '__main__' or module is None:
            continue
        try:
            if _getattribute(module, name, allow_qualname) is obj:
                return module_name
        except AttributeError:
            pass
    return '__main__'
...

= History =
The iteration 

     for module_name, module in sys.modules.items():

in pickle.py was changed in 2007 to

     for name, module in list(sys.modules.items()):

by the BDFL [http://hg.python.org/cpython/rev/0d2dc3611e3b?revcount=800] to "Fix a bizarre error...", then it was reverted by [http://hg.python.org/cpython/rev/992ef855b3ed] in 2013. I hope the BDFL never discover this :-)


= Explanation = 
It seems that while iterating in sys.modules, getattr is called on an instance of scipy.lib.six.MovedModule, which triggers a change in sys.modules (due to an import). This result in the RuntimeError: dictionary changed size during iteration.
Freezing the list, with list(sys.modules.items()) resolves this particular issue.


= Notes on reproducing this error =
Reproducing my test case might be difficult for the following reason.
I installed scipy version 0.13.3 on Kubuntu 14.04 (sudo apt-get install python3-scipy). In the online  source code, module six.py version 1.2.0 is included [https://github.com/scipy/scipy/releases/tag/v0.13.3]. However in my system the module six.py that is actually used is 1.5.2, which probably has been installed independently from scipy. 
Finally, the new version of scipy, 0.14.0, doesn't use the class scipy.lib.six.MovedModule which triggers this error. Unfortunately this new version isn't available in Ubuntu 14.04.
msg226013 - (view) Author: Attilio Di Nisio (attilio.dinisio) * Date: 2014-08-28 09:13
A simple patch to freeze the list of sys.modules.
msg226014 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-08-28 09:18
> A simple patch to freeze the list of sys.modules.

Which patch?
msg226015 - (view) Author: Attilio Di Nisio (attilio.dinisio) * Date: 2014-08-28 09:20
diff -r fb3aee1cff59 Lib/pickle.py
--- a/Lib/pickle.py	Wed Aug 27 09:41:05 2014 -0700
+++ b/Lib/pickle.py	Thu Aug 28 10:59:37 2014 +0200
@@ -280,7 +280,7 @@
     module_name = getattr(obj, '__module__', None)
     if module_name is not None:
         return module_name
-    for module_name, module in sys.modules.items():
+    for module_name, module in list(sys.modules.items()):
         if module_name == '__main__' or module is None:
             continue
         try:
msg228682 - (view) Author: Attilio Di Nisio (attilio.dinisio) * Date: 2014-10-06 13:45
Duplicate of #21905
Fixed in https://hg.python.org/cpython/rev/86ba3bdfac15
History
Date User Action Args
2014-10-06 15:10:31vstinnersetsuperseder: RuntimeError in pickle.whichmodule when sys.modules if mutated
2014-10-06 13:45:50attilio.dinisiosetstatus: open -> closed
resolution: duplicate
messages: + msg228682

versions: + Python 3.5
2014-08-28 20:01:07attilio.dinisiosetfiles: + pickle.py.diff
keywords: + patch
2014-08-28 09:20:02attilio.dinisiosetmessages: + msg226015
2014-08-28 09:18:05vstinnersetnosy: + vstinner
messages: + msg226014
2014-08-28 09:13:19attilio.dinisiosetmessages: + msg226013
2014-08-27 21:51:47attilio.dinisiocreate