Exceptions during import now display huge tracebacks across importlib._bootstrap, this adds a lot of noise to the error:

For example, I added some syntax error in distutils/spawn.py, then:

~/python/cpython3.x$ ./python -c "from distutils import ccompiler"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 1335, in _handle_fromlist
  File "<frozen importlib._bootstrap>", line 1288, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1255, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 778, in load_module
  File "<frozen importlib._bootstrap>", line 759, in load_module
  File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 647, in _load_module
  File "/home/amauryfa/python/cpython3.x/Lib/distutils/ccompiler.py", line 8, in <module>
    from distutils.spawn import spawn
  File "<frozen importlib._bootstrap>", line 1288, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1255, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 778, in load_module
  File "<frozen importlib._bootstrap>", line 759, in load_module
  File "<frozen importlib._bootstrap>", line 408, in module_for_loader_wrapper
  File "<frozen importlib._bootstrap>", line 635, in _load_module
  File "<frozen importlib._bootstrap>", line 736, in get_code
  File "/home/amauryfa/python/cpython3.x/Lib/distutils/spawn.py", line 14
    does not compile
                   ^
SyntaxError: invalid syntax

importlib is written in python.  So you get a python traceback of its execution stack.  Yes it is noisy, but I'm not sure that this should be changed, or we'd lose some of the benefit of having importlib written in python.  (It also might be really complicated to change...I'm sure Brett will tell us :)

I agree that this is not helpful at all in the usual case, i.e. when you *don't* want to debug importlib.  The one frame in actual user code (distutils in this case) in the middle is kind of hard to spot, but it is what you want to know.  Note that Amaury's example is quite small; in other projects the import chains may go on for 10-20 modules until the ImportError is raised.  Good luck finding out where the bad module was imported without cursing.

This is different from the normal case in user code calling stdlib code, which raises an exception: the user code frames will be near the top, and the stdlib code near the bottom.

I see several options here, in no particular order of preference:

1. special-case importlib._bootstrap when printing tracebacks
2. rewrite the traceback object when raising ImportErrors from importlib
5. (is there more?)

Here is a patch which removes "<frozen importlib._bootstrap>" frames from tracebacks.  Tests are missing, but this gives good result on the few samples I tried.
"python -v" does not suppress importlib frames.

Wouldn't it be better to do it in pure Python in importlib itself?

I tried to do it in importlib, with code like 
    raise original_exception.with_traceback(trimmed_traceback)
but the function containing this very line would always be part of the displayed stack trace.

Perhaps at least the trimming should only be done for ImportErrors, then?

No, the example above is a SyntaxError, and I claim the importlib frames should be removed as well.

Ah, you're right. But it could become confusing if some error is raised inside of importlib itself (not an ImportError, something else - perhaps a bug).

So I would suggest the following:
- trim ImportErrors
- trim errors raised when executing the code of an imported module (i.e. in _LoaderBasics._load_module())
- don't trim other errors, since they can be importlib programming errors

About this proposal:

> - trim errors raised when executing the code of an imported module (i.e. in _LoaderBasics._load_module())

The exec() could be isolated in a distinctly-named submethod (e.g. _exec_module()), so that it is easy to detect it in a traceback.

Setting to blocker for b2.

Here is a patch with tests. Brett, what do you think?

By the way, it has to be done in C since tb_next can't be modified from Python code.

Does it work for extension modules?

> Does it work for extension modules?

What do you mean?

I added to _ssl.c:

    PyErr_SetString(PyExc_ValueError, "Just a test");
    return NULL;

Then I tried to import the module:

~/python/cpython3.x$ ./python -c "import ssl"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/amauryfa/python/cpython3.x/Lib/ssl.py", line 60, in <module>
    import _ssl             # if we can't import it, let the error propagate
  File "<frozen importlib._bootstrap>", line 1300, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1267, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 432, in _check_name_wrapper
  File "<frozen importlib._bootstrap>", line 347, in set_package_wrapper
  File "<frozen importlib._bootstrap>", line 360, in set_loader_wrapper
  File "<frozen importlib._bootstrap>", line 878, in load_module
ValueError: Just a test

See also msg164937

Ah, right, here is an updated patch which also handles extension modules and frozen ones.

Looks good to me. Go ahead and commit it.

New changeset 8c877ad00bc4 by Antoine Pitrou in branch 'default':
Issue #15110: Fix the tracebacks generated by "import xxx" to not show the importlib stack frames.
http://hg.python.org/cpython/rev/8c877ad00bc4

Thanks, Brett!

I really like the "_exec_module" trick, but it should be applied to builtin modules as well. I hacked _sre.c and got:

~/python/cpython3.x$ ./python 
Traceback (most recent call last):
  File "/home/amauryfa/python/cpython3.x/Lib/site.py", line 70, in <module>
    import re
  File "/home/amauryfa/python/cpython3.x/Lib/re.py", line 122, in <module>
    import sre_compile
  File "/home/amauryfa/python/cpython3.x/Lib/sre_compile.py", line 13, in <module>
    import _sre, sys
  File "<frozen importlib._bootstrap>", line 1318, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1285, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 347, in set_package_wrapper
  File "<frozen importlib._bootstrap>", line 360, in set_loader_wrapper
  File "<frozen importlib._bootstrap>", line 443, in _requires_builtin_wrapper
  File "<frozen importlib._bootstrap>", line 493, in load_module
ValueError: Just a test


This change correctly hides importlib frames:


diff -r 9afdd8c25bf2 Lib/importlib/_bootstrap.py
--- a/Lib/importlib/_bootstrap.py	Sun Jul 08 14:00:06 2012 +0200
+++ b/Lib/importlib/_bootstrap.py	Sun Jul 08 15:03:27 2012 +0200
@@ -490,12 +490,15 @@
         """Load a built-in module."""
         is_reload = fullname in sys.modules
         try:
-            return _imp.init_builtin(fullname)
+            return self._exec_module(fullname)
         except:
             if not is_reload and fullname in sys.modules:
                 del sys.modules[fullname]
             raise
 
+    def _exec_module(self, fullname):
+        return _imp.init_builtin(fullname)
+
     @classmethod
     @_requires_builtin
     def get_code(cls, fullname):

Re-opening so Antoine can look at Amaury's proposed fix for builtin modules.

> I really like the "_exec_module" trick, but it should be applied to
> builtin modules as well. I hacked _sre.c and got:

I hadn't thought about this one. Can you apply your patch?

New changeset 37e68da59047 by Amaury Forgeot d'Arc in branch 'default':
Issue #15110: Also hide importlib frames when importing a builtin module fails.
http://hg.python.org/cpython/rev/37e68da59047

New changeset 5d43154d68a8 by Amaury Forgeot d'Arc in branch 'default':
Issue #15110: Copy same docstring as other '_exec_module' methods.
http://hg.python.org/cpython/rev/5d43154d68a8

Issue15425 is related.  I'm looking into an exception chaining approach that could be applied for this issue too.