Index: Misc/NEWS =================================================================== --- Misc/NEWS (revision 72884) +++ Misc/NEWS (working copy) @@ -470,6 +470,9 @@ - unittest.assertNotEqual() now uses the inequality operator (!=) instead of the equality operator. + +- Issue #6001: Test discovery for unittest. Implemented in + unittest.TestLoader.discover and from the command line. - Issue #5679: The methods unittest.TestCase.addCleanup and doCleanups were added. addCleanup allows you to add cleanup functions that will be called Index: Doc/library/unittest.rst =================================================================== --- Doc/library/unittest.rst (revision 72884) +++ Doc/library/unittest.rst (working copy) @@ -91,6 +91,9 @@ `python-mock `_ and `minimock `_ Tools for creating mock test objects (objects simulating external resources). + +.. _unittest-command-line-interface: + Command Line Interface ---------------------- @@ -112,10 +115,49 @@ python -m unittest -h -.. versionchanged:: 27 +.. versionchanged:: 2.7 In earlier versions it was only possible to run individual test methods and not modules or classes. +The command line can also be used for test discovery, for running all of the tests in a +project or just a subset. + + +.. _unittest-test-discovery: + +Test Discovery +-------------- + +unittest supports simple test discovery. For a project's tests to be compatible with +test discovery they must all be importable from the top level directory of the project; +i.e. they must all be in Python packages. + +Test discovery is implemented in :meth:`TestLoader.discover`, but can also be used from +the command line. The basic command line usage is:: + + cd project_directory + python -m unittest discover + +The ``discover`` sub-command has the following options:: + + -v, --verbose Verbose output + -s directory Directory to start discovery ('.' default) + -p pattern Pattern to match test files ('test*.py' default) + -t directory Top level directory of project (default to + start directory) + +The -s, -p, & -t options can be passsed in as positional arguments. The following +two command lines are equivalent:: + + python -m unittest -s project_directory -p '*_test.py' + python -m unittest project_directory '*_test.py' + +Test modules and packages can customize test loading and discovery by through the +`load_tests protocol`_. + + +.. versionadded:: 2.7 + .. _unittest-minimal-example: Basic example @@ -1160,6 +1202,13 @@ methods on base classes that are not intended to be instantiated directly does not play well with this method. Doing so, however, can be useful when the fixtures are different and defined in subclasses. + + If a module provides a ``load_tests`` function it will be called to + load the tests. This allows modules to customize test loading. + This is the `load_tests protocol`_. + + .. versionchanged:: 2.7 + Support for ``load_tests`` added. .. method:: loadTestsFromName(name[, module]) @@ -1199,6 +1248,29 @@ Return a sorted sequence of method names found within *testCaseClass*; this should be a subclass of :class:`TestCase`. + + .. method:: discover(start_dir, pattern='test*.py', top_level_dir=None) + + Find and return all test modules from the specified start directory, recursing + into subdirectories to find them. Only test files that match the pattern will + be loaded. (Using glob style pattern matching.) + + All test modules must be importable from the top level of the project. If + the start directory is not the top level directory then the top level + directory must be specified separately. + + If a test package name (directory with '__init__.py') matches the include + filter then the package will be checked for a 'load_tests' function. If + this exists then it will be called with loader, None, include_filter. + + If load_tests exists then discovery does *not* recurse into the package, + load_tests is responsible for loading all tests in the package. + + The include_filter is deliberately not stored as a loader attribute so that + packages can continue discovery themselves. top_level_dir is stored so + load_tests does not need to pass this argument in to ``loader.discover()``. + + The following attributes of a :class:`TestLoader` can be configured either by subclassing or assignment on an instance: @@ -1463,3 +1535,69 @@ .. versionchanged:: 2.7 The ``exit`` and ``verbosity`` parameters were added. + + +load_tests Protocol +################### + +Modules or packages can customize how tests are loaded from them during normal +test runs or test discovery by implementing a function called ``load_tests``. + +If a test module defines ``load_tests`` it will be called by +:meth:`TestLoader.loadTestsFromModule` with the following arguments:: + + load_tests(loader, standard_tests, None) + +It should return a :class:`TestSuite`. + +``loader`` is the instance of :class:`TestLoader` doing the loading. +``standard_tests`` are the tests that would be loaded by default from the +module. It is common for test modules to only want to add or remove tests +from the standard set of tests. +The third argument is used when loading packages as part of test discovery. + +A typical ``load_tests`` function that loads tests from a specific set of +:class:`TestCase` classes may look like:: + + test_cases = (TestCase1, TestCase2, TestCase3) + + def load_tests(loader, tests, pattern): + suite = TestSuite() + for test_class in test_cases: + tests = loader.loadTestsFromTestCase(test_class) + suite.addTests(tests) + return suite + +If discovery is started, either from the command line or by calling +:meth:`TestLoader.discover`, with a pattern that matches a package +name then the package __init__.py will be checked for ``load_tests``. + +.. note:: + + The default pattern is 'test*.py'. This matches all python files + that start with 'test' but *won't* match any test directories. + + A pattern like 'test*' will match test packages as well as + modules. + +If the package __init__.py defines ``load_tests`` then it will be +called and discovery not continued into the package. ``load_tests`` +is called with the following arguments:: + + load_tests(loader, standard_tests, pattern) + +This should return a :class:`TestSuite` representing all the tests +from the package. (``standard_tests`` will only contain tests +collected from __init__.py.) + +Because the pattern is passed into ``load_tests`` the package is +free to continue (and potentially modify) test discovery. A +'do nothing' ``load_tests`` function for a test package would +look like:: + + def load_tests(loader, standard_tests, pattern): + # top level directory cached on loader instance + this_dir = os.path.dirname(__file__) + package_tests = loader.discover(start_dir=this_dir, pattern=pattern) + standard_tests.addTests(package_tests) + return standard_tests Index: Lib/unittest.py =================================================================== --- Lib/unittest.py (revision 72884) +++ Lib/unittest.py (working copy) @@ -56,6 +56,9 @@ import types import warnings +from fnmatch import fnmatch + + ############################################################################## # Exported classes and functions ############################################################################## @@ -1214,6 +1217,7 @@ sortTestMethodsUsing = cmp suiteClass = TestSuite classSuiteClass = ClassTestSuite + _top_level_dir = None def loadTestsFromTestCase(self, testCaseClass): """Return a suite of all tests cases contained in testCaseClass""" @@ -1227,13 +1231,17 @@ testCaseClass) return suite - def loadTestsFromModule(self, module): + def loadTestsFromModule(self, module, use_load_tests=True): """Return a suite of all tests cases contained in the given module""" tests = [] for name in dir(module): obj = getattr(module, name) if isinstance(obj, type) and issubclass(obj, TestCase): tests.append(self.loadTestsFromTestCase(obj)) + + load_tests = getattr(module, 'load_tests', None) + if use_load_tests and load_tests is not None: + return load_tests(self, tests, None) return self.suiteClass(tests) def loadTestsFromName(self, name, module=None): @@ -1302,8 +1310,97 @@ testFnNames.sort(key=_CmpToKey(self.sortTestMethodsUsing)) return testFnNames + def discover(self, start_dir, pattern='test*.py', top_level_dir=None): + """Find and return all test modules from the specified start directory, recursing + into subdirectories to find them. Only test files that match the pattern will + be loaded. (Using glob style pattern matching.) + All test modules must be importable from the top level of the project. If + the start directory is not the top level directory then the top level + directory must be specified separately. + If a test package name (directory with '__init__.py') matches the include + filter then the package will be checked for a 'load_tests' function. If + this exists then it will be called with loader, None, include_filter. + + If load_tests exists then discovery does *not* recurse into the package, + load_tests is responsible for loading all tests in the package. + + The include_filter is deliberately not stored as a loader attribute so that + packages can continue discovery themselves. top_level_dir is stored so + load_tests does not need to pass this argument in to loader.discover(). + """ + if top_level_dir is None and self._top_level_dir is not None: + # make top_level_dir optional if called from load_tests in a package + top_level_dir = self._top_level_dir + elif top_level_dir is None: + top_level_dir = start_dir + + top_level_dir = os.path.abspath(os.path.normpath(top_level_dir)) + start_dir = os.path.abspath(os.path.normpath(start_dir)) + + if not top_level_dir in sys.path: + # all test modules must be importable from the top level directory + sys.path.append(top_level_dir) + self._top_level_dir = top_level_dir + + if start_dir != top_level_dir and not os.path.isfile(os.path.join(start_dir, '__init__.py')): + # what about __init__.pyc or pyo (etc) + raise ImportError('Start directory is not importable: %r' % start_dir) + + tests = list(self._find_tests(start_dir, pattern)) + return self.suiteClass(tests) + + + def _get_module_from_path(self, path): + """Load a module from a path relative to the top-level directory + of a project. Used by discovery.""" + path = os.path.splitext(os.path.normpath(path))[0] + + relpath = os.path.relpath(path, self._top_level_dir) + assert not os.path.isabs(relpath), "Path must be within the project" + assert not relpath.startswith('..'), "Path must be within the project" + + name = relpath.replace(os.path.sep, '.') + __import__(name) + return sys.modules[name] + + def _find_tests(self, start_dir, pattern): + """Used by discovery. Yields test suites it loads.""" + paths = os.listdir(start_dir) + + for path in paths: + full_path = os.path.join(start_dir, path) + # what about __init__.pyc or pyo (etc) + # we would need to avoid loading the same tests multiple times + # from '.py', '.pyc' *and* '.pyo' + if os.path.isfile(full_path) and path.lower().endswith('.py'): + if fnmatch(path, pattern): + # if the test file matches, load it + module = self._get_module_from_path(full_path) + yield self.loadTestsFromModule(module) + elif os.path.isdir(full_path): + if not os.path.isfile(os.path.join(full_path, '__init__.py')): + continue + + load_tests = None + tests = None + if fnmatch(path, pattern): + # only check load_tests if the package directory itself matches the filter + package = self._get_module_from_path(full_path) + load_tests = getattr(package, 'load_tests', None) + tests = self.loadTestsFromModule(package, use_load_tests=False) + + if load_tests is None: + if tests is not None: + # tests loaded from package file + yield tests + # recurse into the package + for test in self._find_tests(full_path, pattern): + yield test + else: + yield load_tests(self, tests, pattern) + defaultTestLoader = TestLoader() @@ -1503,11 +1600,37 @@ # Facilities for running tests from the command line ############################################################################## -class TestProgram(object): - """A command-line program that runs a set of tests; this is primarily - for making test modules conveniently executable. - """ - USAGE = """\ +USAGE_AS_MAIN = """\ +Usage: %(progName)s [options] [tests] + +Options: + -h, --help Show this message + -v, --verbose Verbose output + -q, --quiet Minimal output + +Examples: + %(progName)s test_module - run tests from test_module + %(progName)s test_module.TestClass - run tests from + test_module.TestClass + %(progName)s test_module.TestClass.test_method - run specified test method + +[tests] can be a list of any number of test modules, classes and test +methods. + +Alternative Usage: %(progName)s discover [options] + +Options: + -v, --verbose Verbose output + -s directory Directory to start discovery ('.' default) + -p pattern Pattern to match test files ('test*.py' default) + -t directory Top level directory of project (default to + start directory) + +For test discovery all test modules must be importable from the top +level directory of the project. +""" + +USAGE_FROM_MODULE = """\ Usage: %(progName)s [options] [test] [...] Options: @@ -1522,6 +1645,18 @@ %(progName)s MyTestCase - run all 'test*' test methods in MyTestCase """ + +if __name__ == '__main__': + USAGE = USAGE_AS_MAIN +else: + USAGE = USAGE_FROM_MODULE + + +class TestProgram(object): + """A command-line program that runs a set of tests; this is primarily + for making test modules conveniently executable. + """ + USAGE = USAGE def __init__(self, module='__main__', defaultTest=None, argv=None, testRunner=TextTestRunner, testLoader=defaultTestLoader, exit=True, @@ -1551,6 +1686,10 @@ sys.exit(2) def parseArgs(self, argv): + if len(argv) > 1 and argv[1].lower() == 'discover': + self.do_discovery(argv) + return + import getopt long_opts = ['help','verbose','quiet'] try: @@ -1567,7 +1706,8 @@ return if len(args) > 0: self.testNames = args - if sys.modules['unittest'] is sys.modules['__main__']: + if __name__ == '__main__': + # to support python -m unittest ... self.module = None else: self.testNames = (self.defaultTest,) @@ -1579,6 +1719,62 @@ self.test = self.testLoader.loadTestsFromNames(self.testNames, self.module) + def do_discovery(self, argv, Loader=TestLoader): + # handle command line args for test discovery + argv = argv[2:] + start_dir = None + pattern = None + top_level_dir = None + + lower_args = [entry.lower() for entry in argv] + try: + argv.pop(lower_args.index('-v')) + except ValueError: + pass + else: + self.verbosity = 2 + try: + argv.pop(lower_args.index('--verbose')) + except ValueError: + pass + else: + self.verbosity = 2 + + while argv: + if argv[0].startswith('-'): + break + this = argv.pop(0) + if start_dir is None: + start_dir = this + elif pattern is None: + pattern = this + elif top_level_dir is None: + top_level_dir = this + else: + self.usageExit() + + if len(argv) % 2: + self.usageExit() + + while argv: + (command, value), argv = argv[:2], argv[2:] + if not command.startswith('-'): + self.usageExit() + command = command.lower()[1:] + if command == 's': + start_dir = value + elif command == 'p': + pattern = value + elif command == 't': + top_level_dir = value + else: + self.usageExit() + + start_dir = start_dir or '.' + pattern = pattern or 'test*.py' + loader = Loader() + self.test = loader.discover(start_dir, pattern, top_level_dir) + def runTests(self): if isinstance(self.testRunner, (type, types.ClassType)): try: Index: Lib/test/test_unittest.py =================================================================== --- Lib/test/test_unittest.py (revision 72884) +++ Lib/test/test_unittest.py (working copy) @@ -7,7 +7,9 @@ """ from StringIO import StringIO +import os import re +import sys from test import test_support import unittest from unittest import TestCase, TestProgram @@ -256,6 +258,30 @@ reference = [unittest.TestSuite([MyTestCase('test')])] self.assertEqual(list(suite), reference) + + # Check that loadTestsFromModule honors (or not) a module + # with a load_tests function. + def test_loadTestsFromModule__load_tests(self): + m = types.ModuleType('m') + class MyTestCase(unittest.TestCase): + def test(self): + pass + m.testcase_1 = MyTestCase + + load_tests_args = [] + def load_tests(loader, tests, pattern): + load_tests_args.extend((loader, tests, pattern)) + return tests + m.load_tests = load_tests + + loader = unittest.TestLoader() + suite = loader.loadTestsFromModule(m) + self.assertEquals(load_tests_args, [loader, suite, None]) + + load_tests_args = [] + suite = loader.loadTestsFromModule(m, use_load_tests=False) + self.assertEquals(load_tests_args, []) + ################################################################ ### /Tests for TestLoader.loadTestsFromModule() @@ -3378,6 +3404,286 @@ self.assertEqual(events, expected) +class TestDiscovery(TestCase): + + # Heavily mocked tests so I can avoid hitting the filesystem + def test_get_module_from_path(self): + loader = unittest.TestLoader() + + def restore_import(): + unittest.__import__ = __import__ + unittest.__import__ = lambda *_: None + self.addCleanup(restore_import) + + expected_module = object() + def del_module(): + del sys.modules['bar.baz'] + sys.modules['bar.baz'] = expected_module + self.addCleanup(del_module) + + loader._top_level_dir = '/foo' + module = loader._get_module_from_path('/foo/bar/baz.py') + self.assertEqual(module, expected_module) + + if not __debug__: + # asserts are off + return + + with self.assertRaises(AssertionError): + loader._get_module_from_path('/bar/baz.py') + + def test_find_tests(self): + loader = unittest.TestLoader() + + original_listdir = os.listdir + def restore_listdir(): + os.listdir = original_listdir + original_isfile = os.path.isfile + def restore_isfile(): + os.path.isfile = original_isfile + original_isdir = os.path.isdir + def restore_isdir(): + os.path.isdir = original_isdir + + path_lists = [['test1.py', 'test2.py', 'not_a_test.py', 'test_dir', + 'test.foo', 'another_dir'], + ['test3.py', 'test4.py', ]] + os.listdir = lambda path: path_lists.pop(0) + self.addCleanup(restore_listdir) + + def isdir(path): + return path.endswith('dir') + os.path.isdir = isdir + self.addCleanup(restore_isdir) + + def isfile(path): + # another_dir is not a package and so shouldn't be recursed into + return not path.endswith('dir') and not 'another_dir' in path + os.path.isfile = isfile + self.addCleanup(restore_isfile) + + loader._get_module_from_path = lambda path: path + ' module' + loader.loadTestsFromModule = lambda module: module + ' tests' + + loader._top_level_dir = '/foo' + suite = list(loader._find_tests('/foo', 'test*.py')) + + expected = [os.path.join('/foo', name) + ' module tests' for name in + ('test1.py', 'test2.py')] + expected.extend([os.path.join('/foo', 'test_dir', name) + ' module tests' for name in + ('test3.py', 'test4.py')]) + self.assertEqual(suite, expected) + + def test_find_tests_with_package(self): + loader = unittest.TestLoader() + + original_listdir = os.listdir + def restore_listdir(): + os.listdir = original_listdir + original_isfile = os.path.isfile + def restore_isfile(): + os.path.isfile = original_isfile + original_isdir = os.path.isdir + def restore_isdir(): + os.path.isdir = original_isdir + + directories = ['a_directory', 'test_directory', 'test_directory2'] + path_lists = [directories, [], [], []] + os.listdir = lambda path: path_lists.pop(0) + self.addCleanup(restore_listdir) + + os.path.isdir = lambda path: True + self.addCleanup(restore_isdir) + + os.path.isfile = lambda path: os.path.basename(path) not in directories + self.addCleanup(restore_isfile) + + class Module(object): + paths = [] + load_tests_args = [] + + def __init__(self, path): + self.path = path + self.paths.append(path) + if os.path.basename(path) == 'test_directory': + def load_tests(loader, tests, pattern): + self.load_tests_args.append((loader, tests, pattern)) + return 'load_tests' + self.load_tests = load_tests + + def __eq__(self, other): + return self.path == other.path + + loader._get_module_from_path = lambda path: Module(path) + def loadTestsFromModule(module, use_load_tests): + if use_load_tests: + raise self.failureException('use_load_tests should be False for packages') + return module.path + ' module tests' + loader.loadTestsFromModule = loadTestsFromModule + + loader._top_level_dir = '/foo' + # this time no '.py' on the pattern so that it can match + # a test package + suite = list(loader._find_tests('/foo', 'test*')) + + # We should have loaded tests from the test_directory package by calling load_tests + # and directly from the test_directory2 package + self.assertEqual(suite, ['load_tests', '/foo/test_directory2 module tests']) + self.assertEqual(Module.paths, [os.path.join('/foo', 'test_directory'), + os.path.join('/foo', 'test_directory2')]) + + # load_tests should have been called once with loader, tests and pattern + self.assertEqual(Module.load_tests_args, + [(loader, os.path.join('/foo', 'test_directory') + ' module tests', + 'test*')]) + + def test_discover(self): + loader = unittest.TestLoader() + + original_isfile = os.path.isfile + def restore_isfile(): + os.path.isfile = original_isfile + + os.path.isfile = lambda path: False + self.addCleanup(restore_isfile) + + full_path = os.path.abspath(os.path.normpath('/foo')) + def clean_path(): + if sys.path[-1] == full_path: + sys.path.pop(-1) + self.addCleanup(clean_path) + + with self.assertRaises(ImportError): + loader.discover('/foo/bar', top_level_dir='/foo') + + self.assertEqual(loader._top_level_dir, full_path) + self.assertIn(full_path, sys.path) + + os.path.isfile = lambda path: True + _find_tests_args = [] + def _find_tests(start_dir, pattern): + _find_tests_args.append((start_dir, pattern)) + return ['tests'] + loader._find_tests = _find_tests + loader.suiteClass = str + + suite = loader.discover('/foo/bar/baz', 'pattern', '/foo/bar') + + top_level_dir = os.path.abspath(os.path.normpath('/foo/bar')) + start_dir = os.path.abspath(os.path.normpath('/foo/bar/baz')) + self.assertEqual(suite, "['tests']") + self.assertEqual(loader._top_level_dir, top_level_dir) + self.assertEqual(_find_tests_args, [(start_dir, 'pattern')]) + + def test_command_line_handling_parseArgs(self): + # Haha - take that uninstantiable class + program = object.__new__(TestProgram) + + args = [] + def do_discovery(argv): + args.extend(argv) + program.do_discovery = do_discovery + program.parseArgs(['something', 'discover']) + + self.assertEqual(args, ['something', 'discover']) + + def test_command_line_handling_do_discovery_fails(self): + class Stop(Exception): + pass + def usageExit(): + raise Stop + + program = object.__new__(TestProgram) + program.usageExit = usageExit + + base = ['program', 'discover'] + with self.assertRaises(Stop): + # too many args + program.do_discovery(base + ['one', 'two', 'three', 'four']) + + with self.assertRaises(Stop): + # odd number + program.do_discovery(base + ['-s', 'two', 'three']) + + with self.assertRaises(Stop): + # arg 'two' doesn't start with '-' + program.do_discovery(base + ['-s', 'one', 'two', 'three']) + + with self.assertRaises(Stop): + # unrecognised option + program.do_discovery(base + ['-S', 'one', '-X', 'two']) + + def test_command_line_handling_do_discovery_calls_loader(self): + program = object.__new__(TestProgram) + base = ['program', 'discover'] + + class Loader(object): + args = [] + def discover(self, start_dir, pattern, top_level_dir): + self.args.append((start_dir, pattern, top_level_dir)) + return 'tests' + + program.do_discovery(base + ['-v'], Loader=Loader) + self.assertEqual(program.verbosity, 2) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('.', 'test*.py', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['--verbose'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('.', 'test*.py', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base, Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('.', 'test*.py', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['fish'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('fish', 'test*.py', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['fish', 'eggs'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('fish', 'eggs', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['fish', 'eggs', 'ham'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('fish', 'eggs', 'ham')]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['-S', 'fish'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('fish', 'test*.py', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['-t', 'fish'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('.', 'test*.py', 'fish')]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['-p', 'fish'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('.', 'fish', None)]) + + Loader.args = [] + program = object.__new__(TestProgram) + program.do_discovery(base + ['-p', 'eggs', '-S', 'fish', '-v'], Loader=Loader) + self.assertEqual(program.test, 'tests') + self.assertEqual(Loader.args, [('fish', 'eggs', None)]) + self.assertEqual(program.verbosity, 2) + + ###################################################################### ## Main ###################################################################### @@ -3386,7 +3692,7 @@ test_support.run_unittest(Test_TestCase, Test_TestLoader, Test_TestSuite, Test_TestResult, Test_FunctionTestCase, Test_TestSkipping, Test_Assertions, TestLongMessage, - Test_TestProgram, TestCleanUp) + Test_TestProgram, TestCleanUp, TestDiscovery) if __name__ == "__main__": test_main()