classification
Title: importlib does not properly remove frames when invoking external import hooks
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.4, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, eric.snow, ncoghlan, raulcd, shiz
Priority: normal Keywords: patch

Created on 2015-03-25 03:28 by shiz, last changed 2016-09-08 17:28 by brett.cannon.

Files
File name Uploaded Description Edit
cpython-3ac58de829ef-fix-external-module-loader-call-traceback.patch shiz, 2015-03-25 03:28
Messages (3)
msg239219 - (view) Author: Shiz (shiz) * Date: 2015-03-25 03:28
When adding a custom module loader to sys.meta_path, importlib does not properly remove its frames before invoking it. This results in a weird traceback with importlib._bootstrap frames in if an error occurs during load_module(), like such:

Traceback (most recent call last):
  File "/Users/mark/Development/Projects/rave/rave/bootstrap/__init__.py", line 102, in bootstrap_game
    __import__(MODULE_PACKAGE + '.' + mod.replace('.py', ''))
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/__init__.py", line 1, in <module>
    from . import common, core3
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/core3/__init__.py", line 11, in <module>
    from .texture import Texture, Image
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/core3/texture.py", line 8, in <module>
    raise ValueError

Attached is a patch against the current hg head that makes sure module loaders are invoked through _call_with_frames_removed, so that the output becomes a lot more readable:

Traceback (most recent call last):
  File "/Users/mark/Development/Projects/rave/rave/bootstrap/__init__.py", line 102, in bootstrap_game
    __import__(MODULE_PACKAGE + '.' + mod.replace('.py', ''))
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/__init__.py", line 1, in <module>
    from . import common, core3
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/core3/__init__.py", line 11, in <module>
    from .texture import Texture, Image
  File "/Users/mark/Development/Projects/rave/rave/modularity.py", line 23, in exec_module
    super().exec_module(module)
  File "/.modules/opengl/core3/texture.py", line 8, in <module>
    raise ValueError
ValueError

Ideally it would remove the calls /within/ the module loader itself too, but I do not see an easy way to do this without changing the entire _call_with_frames_removed semantics. This fixes most of the issues, though.
msg240764 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2015-04-13 21:33
Did you happen to run the test suite with this change, Shiz? There's actually a test to make sure that the part of the traceback you're avoiding isn't lost in case there's a bug in importlib itself.

Can you show that your change won't actually affect finding errors in importlib itself?
msg240781 - (view) Author: Shiz (shiz) * Date: 2015-04-13 22:14
Ah, that is a very good point. I'll run the suite as soon as I find the time and will report back.
History
Date User Action Args
2016-09-08 17:28:36brett.cannonsetassignee: brett.cannon ->
2015-04-13 22:14:31shizsetmessages: + msg240781
2015-04-13 21:36:27raulcdsetnosy: + raulcd
2015-04-13 21:33:27brett.cannonsetnosy: + ncoghlan, eric.snow
messages: + msg240764
2015-03-25 13:23:57brett.cannonsetassignee: brett.cannon
2015-03-25 06:57:57ned.deilysetnosy: + brett.cannon

stage: patch review
2015-03-25 03:28:58shizcreate