Need --help, --list options and TestSute::testcase args for test_support.run_unittest


User story:
I am covering CGIHTTPServer with tests according to http://www.rfc-editor.org/rfc/rfc3875.txt speccy and want to run a single test in a cycle. I am running test cases like:

> python test_httpservers.py

I want to get all available tests in file and execute only the one, but test_support.run_unittest doesn't accept any params.

> python test_httpservers.py --help

just runs all the tests.


I propose to:

1. add list option
-l, --list    list the names of test to be executed and exit

> python test_httpservers.py --list
BaseHTTPServer::command
BaseHTTPServer::handler
CGIHTTPServerTestCase::authorization

I omit `test_` prefix and `TestCase` suffixes as they don't hold any value and just clutter the console. My idealistic vision thinks it is better this way.

2. add arguments of test names with wildcards
> python test_httpservers.py --list BaseHTTP*
BaseHTTPServer::command
BaseHTTPServer::handler
> python test_httpservers.py --list *auth*
CGIHTTPServerTestCase::authorization

3. The final goal is to be able to run single test as:
> python test_httpservers.py BaseHTTPServer::handler
CGIHTTPServer::url_collapse_path_split ... ok

This format will allow to copy/paste single test name for easy execution, which is impossible with current format:
test_url_collapse_path_split (__main__.CGIHTTPServerTestCase) ... ok


One step to make stdlib module debug more intuitive.

Damn. I still failed to forge last use case. In should of course read:

> python test_httpservers.py CGIHTTPServer::url_collapse_path_split
CGIHTTPServer::url_collapse_path_split ... ok

Wave develops bad habits.

These sound more like features for the unittest runner (one of which is implemented already).  Also, please don't propagate "::" as a namespace separator in Python.  That's what "." is for.

Yes, running individual tests has already been implemented as part of unittest (where it belongs):

./python -m unittest test.test_httpservers.BaseHTTPServerTestCase.test_handler
.
----------------------------------------------------------------------
Ran 1 test in 0.102s

OK

The 'TestCase' marks classes that are actually test cases rather than helper classes, and test_ is how unittest determines which methods are test methods and which are support methods.  These strings are not fixed from test file to test file, so it is necessary to specify the full name (and should be).

the --list option that prints the test name in a cut and paste friendly fashion is somewhat interesting, perhaps you could open a new issue to propose adding that to unittest.  If you do open an issue, make michael.foord nosy on it.

RDM, running your way will execute test from standard library location of installed Python.

> python -m unittest test.test_httpservers.BaseHTTPServerTestCase.test_handler

But I want to execute test from my own patched copy of test_httpservers.py with whatever python I want. I.e.

> C:\Python27\python.exe Z:\python-cgi-tests\test_httpservers.py ...

I prefer to explicitly specify it in command line to be sure that I execute the right test suite. I also don't feel like typing `-m unittest test.test_httpservers.` each time when there is shell completion, and every decent file manager has keyboard shortcuts for inserting active filename into command line prompt.

You know - I am not a core contributor to maintain a bunch of batch files for all kind of operations, so I am interested to make testing process more intuitive.

I know how "TestCase" and "test_" auto discovery works. The point is to remove the need in extra typing, because test runner can add these suffix/prefixes automatically when parsing command line.

@exarkun: Ok. `.` is fine - `::` was a copy paste from my old PHP test runner.

"""But I want to execute test from my own patched copy of test_httpservers.py with whatever python I want. I.e.

> C:\Python27\python.exe Z:\python-cgi-tests\test_httpservers.py ...
"""

If you use Python 2.7 then the following at the end of the test module enables the same command line features that David Murray pointed you to:

if __name__ == '__main__':
    unittest.main()

If you are using earlier versions of Python you can use unittest2 (which also works with Python 2.7).

> If you use Python 2.7 then the following at the end of the test module enables the same command line features that David Murray pointed you to:
>
> if __name__ == '__main__':
>    unittest.main()

Unfortunately, test_httpservers.py from stdlib contains this code:

def test_main(verbose=None):
    try:
        cwd = os.getcwd()
        test_support.run_unittest(BaseHTTPServerTestCase,
                                SimpleHTTPServerTestCase,
                                CGIHTTPServerTestCase
                                )
    finally:
        os.chdir(cwd)

if __name__ == '__main__':
    test_main()

Right. That code should change. My intention is to go through the whole of the Python test suite and make them compatible with the new unittest features.

I also intend to add extensions to unittest to allow listing of collecting tests and filtering tests by name (at the moment test filtering is by filename only which is useful but not enough)... The first task is to add extension / plugin machinery to unittest and then these use cases become easy - so I will not implement them on their own in unittest until the plugin system is done.

At the same time I don't think there is any point (although I wouldn't argue with) in adding code to do this directly to test_support when it is going to be added to unittest anyway.

Cool, unittest.main() worked for me, but I still lack the introspection to see what tests are available inside the file. Would there be any objections against quickly adding -l, --list option to expose the result of test discovery as it works now?

What are requirements for the old test_support.run_unittest code to reach unittest compatibility? Can't you just deprecate test_support.run_unittest() so maintainers get signal to upgrade their tests suite? Then the upgrade happen faster.


P.S. I do not need many plugin use cases right now. The test suite tools are often separated into three components - test discovery, test runner and test formatting, but that's all.

s/need/have/

Closing again, following Michael’s comment.

Re: run_unittest, it’s a very small function mostly obsolete now, I don’t think there are compat rules.  Just don’t use it.