This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Unittest CLI does not support test packages very well
Type: enhancement Stage: needs patch
Components: Versions: Python 3.3
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, ezio.melotti, michael.foord, r.david.murray
Priority: normal Keywords:

Created on 2012-06-05 21:20 by r.david.murray, last changed 2022-04-11 14:57 by admin.

Messages (3)
msg162374 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-05 21:20
Suppose you have a test package:

  test_pkg
     __init__.py
     test_mytest.py

If __init__.py is empty and you run

   python -m unittest test_pk

no tests are found.

You can get this to work by adding the following boiler plate to __init__.py:

  def load_tests(loader, standard_tests, pattern):
    this_dir = os.path.dirname(__file__)
    if pattern is None:
        pattern = "test*"
    package_tests = loader.discover(start_dir=this_dir,
                                    pattern=pattern,
                                    top_level_dir=this_dir)
    standard_tests.addTests(package_tests)
    return standard_tests

Note that top_level_dir is required to handle specifying more than one test package at a time on the unittest command line.  Otherwise the second package gets a loader that already has _top_level_dir set, and so it fails to default to start_dir.  I suspect this is also a bug.

This works; it uses discovery to find the tests and returns them using the load test protocol.  Other methods could be used to construct the test to add as well.  But all have the serious disadvantage that the package name does not appear in the output.  Running the above test_pkg command line give results like this with -v:

  test_something (test_mytest.Test) ... ok

test_pkg is not mentioned.  This is merely annoying when running a single test package, but if you
do something like:

  python -m unittest -v test_pkg test_pkg2

You can't tell in the verbose output or the test failure output which
test package the tests are from.

In summary, unittest needs better support for test packages.
msg162379 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2012-06-05 22:49
Whilst I agree in principle... The trouble is that when you do this:

    python -m unittest test_pk

What you are saying is "run all the tests from the test_pk module". You *aren't* launching discovery. 

This should work:

    python -m unittest discover -t . test_pkg

This is more verbose than is ideal. Suggestions for improvements welcome. Having unittest revert to discovery when it is passed a package name that turns out to be empty seems a bit magical (and complex in terms of implementation).

Yes, calling loader.discover inside a load_tests function will mutate that loader - so having discover restore _top_level_dir on exit would be better. Can you post that as a separate issue?

I think there is a separate issue for improving the test failure name (including module) reporting. I'll try and dig out the issue number.
msg162385 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2012-06-06 01:15
Right, I'm not wanting to run discovery from the command line, I'm wanting to run the tests in the package by package name.  In my mind, this is exactly parallel to specifying a module name and having unittest automatically discover the TestCase classes in it.  We don't have unittest run 0 tests because discovery wasn't invoked when the module name was specified.  Why should it be different for a test package?  If boilerplate is required in __init__.py to make that happen that's OK, though to my mind not ideal.

Is there some different magic I can put into __init__.py that will result in the tests in the package being run such that the package name shows up in the report?  Without that, specifying a package name on the unittest command line seems pretty useless.  (I mean, to get it to do anything useful, you'd have to be putting all the TestCases in the __init__.py, and if you are doing that, why have a package?)

The issue about improving the name output was about making it copy and pasteable (something I would also very much like).  The naming issue here is different, about how to get the package name to show up in the fully qualified test name.

I will open another bug for the _top_level_dir issue.
History
Date User Action Args
2022-04-11 14:57:31adminsetgithub: 59212
2012-06-15 08:06:10ezio.melottisetnosy: + ezio.melotti
2012-06-08 17:57:42eric.araujosetnosy: + eric.araujo
2012-06-06 01:15:35r.david.murraysetmessages: + msg162385
2012-06-05 22:49:15michael.foordsetmessages: + msg162379
2012-06-05 21:20:28r.david.murraycreate