Say I have a test module test_foo, which fails to import with
ImportError. A reason for this might be a misspelt import in that module.

TestLoader().loadTestsFromName swallows the import error and instead
crashes with:
  File "/usr/lib/python2.6/unittest.py", line 584, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_foo'

A better thing to do would be to keep the import error and if the next
probe is an Attribute error, reraise the import error.
An alternative would be to return a test which would then reraise the
import error permitting the test suite to be constructed and execute but
still reporting the error.

Could you provide a more complete recipe for reproducing the problem,
please?  I created a test_foo.py containing 'import blert', and running
python -m unittest test_foo does not mask the import error for blert in
loadTestsFromName:

...
  File "/usr/lib/python2.6/unittest.py", line 576, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "test_foo.py", line 1, in <module>
    import blert
ImportError: No module named blert

mkdir thing
touch thing/__init__.py
echo "import blert" > thing/test_foo.py
python -m unittest thing.test_fooTraceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/unittest.py", line 875, in <module>
    main(module=None)
  File "/usr/lib/python2.6/unittest.py", line 816, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.6/unittest.py", line 843, in parseArgs
    self.createTests()
  File "/usr/lib/python2.6/unittest.py", line 849, in createTests
    self.module)
  File "/usr/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/usr/lib/python2.6/unittest.py", line 584, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_foo'

Thank you.  I can reproduce this on trunk as well.  I'm leaving stage
set to test needed because we need to turn this into a unit test.

Note: this problem is similar in some ways to issue 5230, and a similar
solution might be appropriate (or might not :).

I'll try and look at both these issues in the next few days unless one
of you beats me to it. :-)

I'm scratching an itch at the moment, I just noted this in passing ;)

I'm partial to the 'turn it into a fake test case' approach, its what I
would do if I get to it first.

Line 348 in trunk/Lib/test/test_unittest.py has a test case to
specifically test that in the described situation, the test returns an
AttributeError. Should this test be changed so that it passes if the
exception is in fact an ImportError?

   def test_loadTestsFromName__unknown_attr_name(self):
        loader = unittest.TestLoader()

        try:
            loader.loadTestsFromName('unittest.sdasfasfasdf')
        except AttributeError, e:
            self.assertEqual(str(e), "'module' object has no attribute
'sdasfasfasdf'")
        else:
            self.fail("TestLoader.loadTestsFromName failed to raise
AttributeError")

I'm attaching a patch (against trunk) which I think is a step in the
right direction but I could use some feedback. 

This patch modifies 'loadTestsFromName()' so that it saves and re-raises
an ImportError.

Further this patch introduces a new unittest
(test_loadTestsFromName__badimport) and slightly modifies two existing
unittests (test_loadTestsFromName__unknown_attr_name,
test_loadTestsFromNames__unknown_attr_name) in test_unittest.py. Also, I
think a second new unittest is needed (test_loadTestsFromNames__badimport).

Hope everyone had a good new year's. I've attached an updated patch which adds a new unittest, test_loadTestsFromNames__badimport. 

Both the new unittests can use better documentation, hopefully one of you can help me with that.

Wouldn't this be a backwards incompatible change of tested behaviour though?

I'm unhappy with a straight change in behaviour because it will break code that is currently catching AttributeError.

A slightly less invasive change would be to raise an AttributeError if the module doesn't exist, otherwise letting the original error propagate.

That means distinguishing between a module not existing and an ImportError raised whilst importing the module. Example code that does this by walking the stack: http://twistedmatrix.com/trac/browser/trunk/twisted/python/reflect.py#L382

In addition we could add a new method that loads a test from name, returning an 'ErrorHolder' if loading the test fails. (A TestCase that reraises the original error when run - test discovery already does this in fact so that a test module failing to load doesn't halt discovery.)

I was also hit by this today.

For the sake of clarity, I will restate two of the scenarios that have been mentioned in this discussion:

(1) An ImportError raised whilst importing a module (original issue)
(2) A sub-module not existing.

I think the error text should be better in both cases and not just in case (1).

Currently, both (1) and (2) yield an error like the following:

AttributeError: 'module' object has no attribute 'subpackage1'

But also in case (2), the AttributeError reveals less information than the exception that was trapped earlier:

ImportError: No module named subpackage1.subpackage2

I think in both cases the error text should state not just what module was being imported but also what module was being imported from -- e.g. root_package.subpackage1.subpackage2.  In other words, it should also include the leading parts of--

'.'.join(parts_copy)  

In my case, I passed a list of modules to unittest, and it wasn't clear which one it was failing on by looking at only the trailing segment.  Thanks.

> I think in both cases the error text should state not just what module was being imported but also what module was being imported from

FYI, I filed the following report partly in response to some of the comments I made above:

http://bugs.python.org/issue8297

(regarding the AttributeError not displaying the name of the module from which the caller is trying to get the attribute)

This patch implements Michael's suggestion (but not the ErrorHolder part):

http://bugs.python.org/issue7559#msg97462

The unit tests all pass with no change.  If this approach looks good to you, I can add a unit test to the patch that checks that this bug has been fixed.

Also, Twisted Matrix's web site doesn't seem to be responding too well at the moment, but if I recall correctly, their code has a permissive (MIT?) license that should allow a small snippet like this to be copied without taking extra steps.

Rietveld link: http://codereview.appspot.com/810044/show

This patch changes unittest.TestLoader.loadTestsFromName() so that ImportErrors will bubble up when importing from a module with a bad import statement.  Before the method raised an AttributeError.  The unit test code is taken from a patch by Salman Haq.  The patch also includes code adapted from http://twistedmatrix.com .

(This is my first patch, so any guidance is greatly appreciated.  Thanks.)

The unit test passes on trunk for me without the fix applied.

Thanks, David.  Sorry about that.  The test probably requires one additional level of nesting so that "parts_copy" is not False:

+                    if not parts_copy or not module_not_found:
                         raise

FYI, there seems to be a bug in the code cited above:

http://twistedmatrix.com/trac/browser/trunk/twisted/python/reflect.py#L382

For example, _importAndCheckStack('package.subpackage.module') raises
_NoModuleFound in the following scenario:

package/subpackage/__init__.py:
   import no_exist

when it should instead raise an ImportError from the buggy __init__.py.

I now think there should be at least a few unit tests to cover this case and a couple similar permutations.

Four failing unit tests (context code can use clean-up).

The new unit tests pass with this patch (minor clean-up still needed).

Thank you very much for those tests.  I think you can simplify them a bit. For example, you can use assertRaises.  You also might be able to use the test_support.temp_cwd context manager in your context manager, even though you don't need the cwd part.

I've attached an alternate, simpler patch to fix this bug, based on a similar fix I did in pydoc.  The disadvantage of my patch is that it contains a hardcoding of the name of the function doing the import.  I think this is acceptable given the much greater simplicity of my patch.  I may be missing some subtlety, though, that the twisted folks know about.  Or perhaps people will just find the hardcoding itself objectionable.

Thanks for your suggestions on the test code.  I will do that.

It seems like the hard-coded approach would be more brittle.  For example, if someone wants to replace __import__ with their own, e.g.

old__import__ = __builtins__.__import__

def __my_logging_import(*args, **kwargs):
    print "Importing %s..." % args[0]  # module name
    return old__import__(*args, **kwargs)

__builtins__.__import__ = __my_logging_import

Then the stack traces would be different:

  File "/Users/chris_g4/dev/Python/trunk/Lib/unittest/loader.py", line 92, in loadTestsFromName
    module = __import__('.'.join(parts_copy))
  File "unittests.py", line 8, in __my_logging_import
    return old__import__(*args, **kwargs)
ImportError: No module named sdasfasfasdf

This causes the unit tests not to pass.

> I think you can simplify them a bit. For example, you can use assertRaises.

Actually, assertRaises doesn't seem to permit checking error text.  That may be one reason why try-except-else is being used instead throughout.

Patch update: added unit test to cover replacing __import__, incorporated R. David Murray's suggestion to use test_support.test_cwd(), and overall code clean-up.

Also uploaded as Patch 3 to--

http://codereview.appspot.com/810044/show

May I just add that I also ran into this and give my +1 for any fix :-)

Michael, if you have no objection to this patch I'm willing to commit it.

My thinking on this has evolved a bit. Changing an import error into an attribute error is just a bad api. We should just fix the bad api.

I'm still getting hit with this.  In what versions is it okay for us to fix the bad API, as Michael suggested?

Does fix for issue 1559549 (and the fact that importlib is in) allow for a cleaner fix for this?  I've been putting off dealing with this issue in the expectation that it would.

To answer the question: the fixed API can go in 3.3.  If we fix this in earlier versions the API should probably stay the same there, and since the above is 3.3 only, the existing patch is probably appropriate there.  

If Michael doesn't pass judgement soon I'll take a look, since this causes me problems at least once a week.

My favoured fix is to catch the exception and generate a failing test that re-raises the *original exception* (with traceback) when run. That way a single failing module doesn't kill a whole test run (although it does mean later feedback about misspelt imports). It also means (the main problem being reported here) that unittest no longer masks exceptions whilst importing test modules.

This would be a new feature / api change - so it would be Python 3.3 only (but it would go into unittest2).

I updated the tests to Python3, and attempted to replicate the fix using the new importlib qualname support.  Even if it had worked, this would not have finished the patch, since Michael wants to generate a failing test instead of raising the import error.

However, I'm running into weird problems and am shelving this for the moment.  The issue is that if I run the tests like this:

   ./python -m unittest test.test_unittest

(or via regrtest) they fail with the wrong name in the error message.  If I run them like this:

    ./python -m unittest unittest.test.test_loader.TestLoader.<name of test>

the right name is in the message.  I suspect the bug is in the tests, but I'm not spotting it.  Maybe someone else will see it.

Thanks.  It looks like the issue with the latest patch is caused by side effects of calling importlib.import_module().

Working from the patch, I got it to the point where inserting the following four lines somewhere in the code--

    try:
        importlib.import_module('foo__doesnotexist')
    except:
        pass

caused the exception raised by the following line--

    module = importlib.import_module('package_foo2.subpackage.no_exist')

to change from this--

  ...
  File "<frozen importlib._bootstrap>", line 1250, in _find_and_load_unlocked
ImportError: No module named 'package_foo2.subpackage.no_exist'

to this--

  ...
  File "..../Lib/importlib/_bootstrap.py", line 1257, in _find_and_load_unlocked
    raise ImportError(_ERR_MSG.format(name), name=name)
ImportError: No module named 'package_foo2'

It looks like this issue is cropping up in the tests because the test code dynamically adds packages to directories that importlib may already have examined.

In the reduced test case I was creating to examine the issue, I found that inserting a call to importlib.invalidate_caches() at an appropriate location resolved the issue.

Should loadTestsFromName() call importlib.invalidate_caches() in the new patch implementation, or should the test code be aware of that aspect of loadTestsFromName()'s behavior and be adjusted accordingly (e.g. by creating the dynamically-added packages in more isolated directories)?  For backwards compatibility reasons, how does loadTestsFromName() currently behave in this regard (i.e. does importlib.import_module() behave the same as __import__ with respect to caching)?

Thanks for figuring that out.  And no, it doesn't matter if it is importlib.load_module or __import__, since both are provided by importlib now and both use the cache.

It's an interesting question where the cache clear should go.  I *think* it should go in the test, based on the idea that the cache is part of the environment, and therefore should be reset by tests that change what's on the path.  I'm not sure how we'd write an environment monitor for that, since not all changes to the import cache need to be reset.  I wonder if it would be worth putting a reset into DirsOnSysPath.

Not sure what DirsOnSysPath is, but I have been only calling importlib.invalidate_caches() as needed in order to not slow down tests needlessly.

And as for detecting an environment change as necessary, that's essentially impossible since it's only needed if something changed between imports which would require adding a hook to notice that an import happened *and* a directory already covered by sys.path_importer_cache (not sys.path since that doesn't cover packages) changed w/o calling invalidate_caches().

OK, let's just do it in the individual test, then.

Because we don't know if the rest of the test code will adhere to this, we might want to consider clearing the cache before each test as well.

Alternatively, we could avoid having to call importlib.invalidate_caches() at all (and having to think about for which tests it is necessary) if we do each test in a different directory and with a different name for the test package.  We could do the former as follows:

    with support.temp_cwd(support.TESTFN):
        dir_name = self.id().split('.')[-1]  # test method name
        with support.temp_cwd(dir_name) as cwd:
            with support.DirsOnSysPath(cwd):
                # Create package and run test.

An approach like this might be less prone to issues that are hard to troubleshoot.  I verified that it works.

That would probably be OK, but I don't see why clearing the cache in those same methods (that create directories on the path) would be any harder.  (It isn't necessary to clear the cache *afterward*, only before, as far as I can see, since the case of a directory not existing that the cache thinks exists should be handled correctly by importlib).

That sounds fine.  I just got the sense from above that there was a desire to call invalidate_caches() as few times as possible.

And yes, I agree only before is necessary.  I had just taken what you said above literally (that "[the cache] should be reset by tests that change what's on the path"), thinking that you wanted to maintain the principle that tests should leave things as they were at the beginning.

Ah, yes, I wasn't clear.  Sorry.

What can I do to put this forward? It's still an issue in py2.7

My preferred fix is to wrap "an exception during import" as a test that fails instead of an AttributeError. This would definitely be a new feature rather than a bugfix - so it could only be in 3.4. 

It could be made available to Python 2.7 through the unittest2 backport.

None of the current patches implement my preferred solution yet.

One relevant use case is the following: https://github.com/Pylons/venusian/issues/23

Here the module is supposed to raise an ImportError.

Note that this issue is referred to from #15358.

Note that #8297 referenced in msg102236 is closed see changeset d84a69b7ba72.

I've just put a patch up for the related issue http://bugs.python.org/issue19746

I'll poke at this one briefly now, since I'm across the related code.

Ok, here is an implementation that I believe covers everything Michael wanted. I examined the other patches, and can rearrange my implementation to be more like them if desired - but at the heart of this this bug really has two requested changes:
 - deferred reporting of error per Michaels request
 - report missing attributes on packages as an ImportError (if one occurred)

and thus my implementation focuses on those changes.

Thanks for tackling this.  It's been bugging me almost daily this past week, but as usual when this bug is in my face I had no time to actually work on a fix.

I applied this patch to default,  put an invalid import in test_os, and this is the result:

rdmurray@pydev:~/python/p35>./python -m unittest test.test_os
Traceback (most recent call last):
  File "/home/rdmurray/python/p35/Lib/unittest/loader.py", line 140, in loadTestsFromName
    module = __import__(module_name)
  File "/home/rdmurray/python/p35/Lib/test/test_os.py", line 5, in <module>
    import foobar
ImportError: No module named 'foobar'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rdmurray/python/p35/Lib/runpy.py", line 170, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/rdmurray/python/p35/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/rdmurray/python/p35/Lib/unittest/__main__.py", line 18, in <module>
    main(module=None)
  File "/home/rdmurray/python/p35/Lib/unittest/main.py", line 92, in __init__
    self.parseArgs(argv)
  File "/home/rdmurray/python/p35/Lib/unittest/main.py", line 139, in parseArgs
    self.createTests()
  File "/home/rdmurray/python/p35/Lib/unittest/main.py", line 146, in createTests
    self.module)
  File "/home/rdmurray/python/p35/Lib/unittest/loader.py", line 202, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/home/rdmurray/python/p35/Lib/unittest/loader.py", line 202, in <listcomp>
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/home/rdmurray/python/p35/Lib/unittest/loader.py", line 145, in loadTestsFromName
    next_attribute, self.suiteClass)
ValueError: need more than 1 value to unpack

I get similar errors if I misspell the module name in on the command line.

From my point of view this is still an improvement over the status quo, but I don't think it is what you had in mind :).

You may need to apply the patch from http://bugs.python.org/issue19746 first as well - I was testing with both applied.

OK, with both patches applied the output looks good.  With a bit of luck I'll have some time to actually review the patches in a couple of hours.

This is what I see in my tree:

E
======================================================================
ERROR: test_os (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/robertc/work/cpython/Lib/unittest/case.py", line 58, in testPartExecutor
    yield
  File "/home/robertc/work/cpython/Lib/unittest/case.py", line 577, in run
    testMethod()
  File "/home/robertc/work/cpython/Lib/unittest/loader.py", line 36, in testFailure
    raise exception
ImportError: Failed to import test module: test_os
Traceback (most recent call last):
  File "/home/robertc/work/cpython/Lib/unittest/loader.py", line 146, in loadTestsFromName
    module = __import__(module_name)
  File "/home/robertc/work/cpython/Lib/test/test_os.py", line 5, in <module>
    import broken
ImportError: No module named 'broken'

Raced with your comment. Great - and thanks!

Patch polished up and updated - ready for review IMO.

Thanks for the review, updated patch here - I'll let this sit for a day or two for more comments then commit it Monday.

I've updated the patch to try and address the niggling clarity issues from the review. Please let me know what you think (and if I hear nothing I'll commit it as-is since the review was still ok).

New changeset 708b2e656c1d by Robert Collins in branch 'default':
Close #7559: ImportError when loading a test now shown as ImportError.
https://hg.python.org/cpython/rev/708b2e656c1d

Could we backport this one to 3.x and 2.7? It's leads to really bad UX.

I would love that, but I think the fix depends on a feature.  Robert will know for sure.

Its backported in unittest2 0.8.0 which is available on pypi for 2.6+ and 3.2+.

The changes are large enough that I'd hesitate to backport them in cPython itself.

It looks to me like this is complete, so closing.