classification
Title: test_support.run_unittest cmdline options and arguments
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: eric.araujo, exarkun, giampaolo.rodola, michael.foord, pitrou, r.david.murray, techtonik
Priority: normal Keywords:

Created on 2010-06-18 13:03 by techtonik, last changed 2010-11-28 04:34 by eric.araujo. This issue is now closed.

Messages (11)
msg108104 - (view) Author: anatoly techtonik (techtonik) Date: 2010-06-18 13:03
Need --help, --list options and TestSute::testcase args for test_support.run_unittest


User story:
I am covering CGIHTTPServer with tests according to http://www.rfc-editor.org/rfc/rfc3875.txt speccy and want to run a single test in a cycle. I am running test cases like:

> python test_httpservers.py

I want to get all available tests in file and execute only the one, but test_support.run_unittest doesn't accept any params.

> python test_httpservers.py --help

just runs all the tests.


I propose to:

1. add list option
-l, --list    list the names of test to be executed and exit

> python test_httpservers.py --list
BaseHTTPServer::command
BaseHTTPServer::handler
CGIHTTPServerTestCase::authorization

I omit `test_` prefix and `TestCase` suffixes as they don't hold any value and just clutter the console. My idealistic vision thinks it is better this way.

2. add arguments of test names with wildcards
> python test_httpservers.py --list BaseHTTP*
BaseHTTPServer::command
BaseHTTPServer::handler
> python test_httpservers.py --list *auth*
CGIHTTPServerTestCase::authorization

3. The final goal is to be able to run single test as:
> python test_httpservers.py BaseHTTPServer::handler
CGIHTTPServer::url_collapse_path_split ... ok

This format will allow to copy/paste single test name for easy execution, which is impossible with current format:
test_url_collapse_path_split (__main__.CGIHTTPServerTestCase) ... ok


One step to make stdlib module debug more intuitive.
msg108105 - (view) Author: anatoly techtonik (techtonik) Date: 2010-06-18 13:05
Damn. I still failed to forge last use case. In should of course read:

> python test_httpservers.py CGIHTTPServer::url_collapse_path_split
CGIHTTPServer::url_collapse_path_split ... ok

Wave develops bad habits.
msg108107 - (view) Author: Jean-Paul Calderone (exarkun) * (Python committer) Date: 2010-06-18 13:20
These sound more like features for the unittest runner (one of which is implemented already).  Also, please don't propagate "::" as a namespace separator in Python.  That's what "." is for.
msg108109 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2010-06-18 13:55
Yes, running individual tests has already been implemented as part of unittest (where it belongs):

./python -m unittest test.test_httpservers.BaseHTTPServerTestCase.test_handler
.
----------------------------------------------------------------------
Ran 1 test in 0.102s

OK

The 'TestCase' marks classes that are actually test cases rather than helper classes, and test_ is how unittest determines which methods are test methods and which are support methods.  These strings are not fixed from test file to test file, so it is necessary to specify the full name (and should be).

the --list option that prints the test name in a cut and paste friendly fashion is somewhat interesting, perhaps you could open a new issue to propose adding that to unittest.  If you do open an issue, make michael.foord nosy on it.
msg108241 - (view) Author: anatoly techtonik (techtonik) Date: 2010-06-20 19:51
RDM, running your way will execute test from standard library location of installed Python.

> python -m unittest test.test_httpservers.BaseHTTPServerTestCase.test_handler

But I want to execute test from my own patched copy of test_httpservers.py with whatever python I want. I.e.

> C:\Python27\python.exe Z:\python-cgi-tests\test_httpservers.py ...

I prefer to explicitly specify it in command line to be sure that I execute the right test suite. I also don't feel like typing `-m unittest test.test_httpservers.` each time when there is shell completion, and every decent file manager has keyboard shortcuts for inserting active filename into command line prompt.

You know - I am not a core contributor to maintain a bunch of batch files for all kind of operations, so I am interested to make testing process more intuitive.

I know how "TestCase" and "test_" auto discovery works. The point is to remove the need in extra typing, because test runner can add these suffix/prefixes automatically when parsing command line.

@exarkun: Ok. `.` is fine - `::` was a copy paste from my old PHP test runner.
msg108242 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-06-20 19:54
"""But I want to execute test from my own patched copy of test_httpservers.py with whatever python I want. I.e.

> C:\Python27\python.exe Z:\python-cgi-tests\test_httpservers.py ...
"""

If you use Python 2.7 then the following at the end of the test module enables the same command line features that David Murray pointed you to:

if __name__ == '__main__':
    unittest.main()

If you are using earlier versions of Python you can use unittest2 (which also works with Python 2.7).
msg108244 - (view) Author: anatoly techtonik (techtonik) Date: 2010-06-20 20:01
> If you use Python 2.7 then the following at the end of the test module enables the same command line features that David Murray pointed you to:
>
> if __name__ == '__main__':
>    unittest.main()

Unfortunately, test_httpservers.py from stdlib contains this code:

def test_main(verbose=None):
    try:
        cwd = os.getcwd()
        test_support.run_unittest(BaseHTTPServerTestCase,
                                SimpleHTTPServerTestCase,
                                CGIHTTPServerTestCase
                                )
    finally:
        os.chdir(cwd)

if __name__ == '__main__':
    test_main()
msg108245 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2010-06-20 20:09
Right. That code should change. My intention is to go through the whole of the Python test suite and make them compatible with the new unittest features.

I also intend to add extensions to unittest to allow listing of collecting tests and filtering tests by name (at the moment test filtering is by filename only which is useful but not enough)... The first task is to add extension / plugin machinery to unittest and then these use cases become easy - so I will not implement them on their own in unittest until the plugin system is done.

At the same time I don't think there is any point (although I wouldn't argue with) in adding code to do this directly to test_support when it is going to be added to unittest anyway.
msg110381 - (view) Author: anatoly techtonik (techtonik) Date: 2010-07-15 17:00
Cool, unittest.main() worked for me, but I still lack the introspection to see what tests are available inside the file. Would there be any objections against quickly adding -l, --list option to expose the result of test discovery as it works now?

What are requirements for the old test_support.run_unittest code to reach unittest compatibility? Can't you just deprecate test_support.run_unittest() so maintainers get signal to upgrade their tests suite? Then the upgrade happen faster.


P.S. I do not need many plugin use cases right now. The test suite tools are often separated into three components - test discovery, test runner and test formatting, but that's all.
msg110382 - (view) Author: anatoly techtonik (techtonik) Date: 2010-07-15 17:01
s/need/have/
msg122626 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-11-28 04:34
Closing again, following Michael’s comment.

Re: run_unittest, it’s a very small function mostly obsolete now, I don’t think there are compat rules.  Just don’t use it.
History
Date User Action Args
2010-11-28 04:34:38eric.araujosetstatus: open -> closed
nosy: + eric.araujo
messages: + msg122626

2010-07-15 17:01:32techtoniksetmessages: + msg110382
2010-07-15 17:00:41techtoniksetmessages: + msg110381
2010-06-20 20:09:26michael.foordsetmessages: + msg108245
2010-06-20 20:01:14techtoniksetmessages: + msg108244
2010-06-20 19:54:30michael.foordsetmessages: + msg108242
2010-06-20 19:51:48techtoniksetstatus: closed -> open
nosy: + pitrou, michael.foord
messages: + msg108241

2010-06-18 17:03:13giampaolo.rodolasetnosy: + giampaolo.rodola
2010-06-18 13:56:57r.david.murraylinkissue9027 superseder
2010-06-18 13:55:12r.david.murraysetstatus: open -> closed

type: enhancement

nosy: + r.david.murray
messages: + msg108109
resolution: out of date
stage: resolved
2010-06-18 13:20:28exarkunsetnosy: + exarkun
messages: + msg108107
2010-06-18 13:05:28techtoniksetmessages: + msg108105
2010-06-18 13:03:41techtonikcreate