classification
Title: importlib reload can fail with AttributeError if module removed from sys.path
Type: Stage:
Components: Versions: Python 3.4, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, mforbes, ncoghlan, ned.deily
Priority: normal Keywords:

Created on 2014-05-31 01:56 by ned.deily, last changed 2018-07-16 00:36 by brett.cannon.

Files
File name Uploaded Description Edit
test_sitecustomize.sh ned.deily, 2014-05-31 01:56
Messages (4)
msg219436 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-31 01:56
There are significant differences in behavior between Python 2.7, 3.3, and 3.4 (or current default) when using import reload() while manipulating sys.path.  These differences cause unexpected behavior in the "Run Script" command of the TextMate 2 editor's Python bundle.  TM2's Python "Run Script" is designed to work with either Python 2 or 2 interpreters.  To do things like capture script stdout and stderr, the bundle launches a Python interpreter while prepending a directory with a custom sitecustomize module to sys.path.  The TM2 sitecustomize then removes its directory from sys.path and uses reload to attempt to import any existing sitecustomize module.  It then proceeds to do its TM2 customizations.  In simplified pseudo-code:

# TM_DIRECTORY/sitecustomize.py
import sys
# remove this copy of sitecustomize from import path
if "TM_DIRECTORY" in sys.path:
    sys.path.remove("TM_DIRECTORY")
try:
    import sitecustomize    # get module reference
    if sys.version_info[0] >= 3:
        from imp import reload
    reload(sitecustomize)   # try to import another
                            # using altered path
except ImportError:
    pass                    # no other sitecustomize found
# do TM2 customizations here
# ...

this seems to work just fine with Python 2.7.  With Python 3.3 the reload causes the originally imported sitecustomize to be used again, regardless of whether another sitecustomize exists on sys.path.  For Python 3.4, 1d67eb1df5a9 for Issue19851 changed reloading.  Now for 3.4 (and default), if another sitecustomize exists on sys.path, the behavior is as expected as in 2.7: the other sitecustomize is imported by reload and executes.  However, if there is no other sitecustomize, 3.4/default now gets an AttributeError exception in reload, rather than the expected ImportError.  Enabling PYTHONVERBOSE shows:

  File "./lib/python3.5/importlib/__init__.py", line 148, in reload
    _bootstrap._exec(spec, module)
  File "<frozen importlib._bootstrap>", line 1083, in _exec
AttributeError: 'NoneType' object has no attribute 'name'

And this is due to the call in reload() to _bootstrap._find_spec() returning None (Lib/importlib/__init__.py:147), presumably since there no longer is a sitecustomize.
It looks like the following patch fixes the problem for this case:

diff --git a/Lib/importlib/__init__.py b/Lib/importlib/__init__.py
--- a/Lib/importlib/__init__.py
+++ b/Lib/importlib/__init__.py
@@ -145,6 +145,9 @@
             pkgpath = None
         target = module
         spec = module.__spec__ = _bootstrap._find_spec(name, pkgpath, target)
+        if not spec:
+            msg = "module {} cannot be reloaded"
+            raise ImportError(msg.format(name), name=name)
         _bootstrap._exec(spec, module)
         # The module may have replaced itself in sys.modules!
         return sys.modules[name]

I'm not sure if this is a correct or sufficient change in the general case.

http://stackoverflow.com/questions/23962319/error-output-in-textmate-2-using-python-3-4
msg321701 - (view) Author: Michael Forbes (mforbes) Date: 2018-07-16 00:25
What is the status of this issue?  Is there a work-around?  The 3.7 codebase raises an exception, but what is the correct solution for reloading a module that was loaded dynamically without modifying `sys.path`?
msg321702 - (view) Author: Michael Forbes (mforbes) Date: 2018-07-16 00:32
It seems that a reasonable workaround (if you generate your own `spec`) is to call `spec.loader.exec_module(mod)`.  This seems to reload the module into the same namespace. (I was afraid that it might replace the namespace hence the desire to use `importlib.reload()` explicitly.)
msg321703 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2018-07-16 00:36
If there's no spec, Michael, then your work-around is the next best option (and reloading never actually replaces the actual module object or it's dict/global namespace because we don't know who is holding on to a reference).
History
Date User Action Args
2018-07-16 00:36:57brett.cannonsetnosy: + ncoghlan
messages: + msg321703
2018-07-16 00:32:18mforbessetmessages: + msg321702
2018-07-16 00:25:31mforbessetnosy: + mforbes
messages: + msg321701
2014-05-31 01:56:36ned.deilycreate