classification
Title: regrtest: add --list-cases option to only display test case identifiers
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, louielu, mdk, michael.foord, rbcollins, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2017-05-31 09:22 by vstinner, last changed 2017-06-27 14:56 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1885 closed vstinner, 2017-05-31 09:26
PR 2238 merged louielu, 2017-06-16 03:45
PR 2243 merged louielu, 2017-06-16 10:38
PR 2244 merged vstinner, 2017-06-16 11:35
PR 2249 merged vstinner, 2017-06-16 15:11
PR 2250 merged vstinner, 2017-06-16 15:29
PR 2401 merged vstinner, 2017-06-26 10:58
PR 2441 merged vstinner, 2017-06-27 13:50
PR 2442 merged vstinner, 2017-06-27 14:15
PR 2444 merged vstinner, 2017-06-27 14:49
Messages (26)
msg294825 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-31 09:22
Attached pull requests adds a --list-tests option to only display the list of test names, don't run tests.

It's a first step to implement the issue #29512: "regrtest refleak: implement bisection feature".
msg294828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-31 09:28
Maybe the feature already exists but I failed to find it. Since I'm suprised that nobody implements that before (in unittest), I wait for a first feedback before starting to document the new feature and to write an unit test for it.

support.run_unittest() doesn't use unittest.main() and so don't support the new --list-tests option.
msg294829 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-31 09:30
I wrote this patch because I'm trying to find which test of test_threading leaks a reference:

./python -m test -R 3:3 test_threading

Usually, I modify Lib/test/test_threading.py to remove tests one by one. I "bisect" the test manually. But this process is slow and painful, and it seems like we have many tests which leak references!
msg294830 - (view) Author: Julien Palard (mdk) * Date: 2017-05-31 09:56
If it can be of any help, pytest have a "pytest --collect-only", it may make sense to use the same option name for consistency. I also though about "--dry-run" but I still prefer --list-tests or --collect-only.
msg294840 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-31 12:28
I used testr which has a --list-tests option:
http://testrepository.readthedocs.io/en/latest/MANUAL.html#listing-tests
msg295084 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-03 15:50
How to use this feature? "./python -m unittest --list-tests" produces nothing.
msg295087 - (view) Author: Julien Palard (mdk) * Date: 2017-06-03 16:42
It works in "discover" mode, I tested:

./python -m unittest discover ./Lib/unittest/test/ --list-tests

and it worked.

So maybe the --list-tests should be moved to _getDiscoveryArgParser?

Also spotted that tests are printed on stderr, typically when the users asks for something, it has to be printed to stdout. Typically usages are written on stderr in case of error, but on stdout when explicitly asked via --help, see:

$ ./python -m unittest discover ./Lib/unittest/test/ --tabayo | wc
usage: python -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b]
                                   [-s START] [-p PATTERN] [-t TOP]
                                   [--list-tests]
python -m unittest discover: error: unrecognized arguments: --tabayo
      0       0       0
$ ./python -m unittest discover ./Lib/unittest/test/ --help | wc
     23     129    1110
msg295146 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-04 22:13
I used ./python -m test Lib/test/test_threading.py --list-tests.

I would like to be able to use this feature in regrtest too:

./python -m test test_threading --list-tests

Sadly, this option already exists and only return test files, not test
methods.
msg295166 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-05 05:07
I concur with Julien, the result should be printed to stdout.

I tried:

$ ./python -m unittest test.test_builtin --list-tests
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
test.test_builtin.BuiltinTest.test_abs
test.test_builtin.BuiltinTest.test_all
...

$ ./python -m test.test_builtin --list-tests
BuiltinTest.test_abs
BuiltinTest.test_all
BuiltinTest.test_any
...
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest

All these doctest.DocTestCase.runTest look useless.

But "./python -m unittest test.test_json --list-tests" and "./python -m test.test_json --list-tests" give the same output.

"./python -m test.test_doctest --list-tests" and "./python -m test.test_tk --list-tests" run tests instead of listing them.
msg295500 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-09 08:43
The main problem is doctests. Numerous "doctest.DocTestCase.runTest" look useless and confusing.

Running tests instead of listing them is bad.

It is not clear why in one cases --list-tests outputs full qualified test names, while in other cases it outputs only class and method names.
msg296158 - (view) Author: Louie Lu (louielu) * Date: 2017-06-16 03:49
Serhiy and Victor suggest moving this option to regrtest. open a new PR 2238 addressed this.

Also, because moving into regrtest, some test that used `requires` will need to pass option when using it. e.g.:

  $ ./python -m test test.test_curses --list-cases -ucurses

otherwise, it will just passout and do nothing (maybe it should warn user?)
msg296170 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 09:36
New changeset a49c935cfd6b4f8ea1f750888dd9260bbf5b9980 by Victor Stinner (mlouielu) in branch 'master':
bpo-30523: regrtest: Add --list-cases option (#2238)
https://github.com/python/cpython/commit/a49c935cfd6b4f8ea1f750888dd9260bbf5b9980
msg296175 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 10:24
Yeah, Louie fixed most issues!

> I concur with Julien, the result should be printed to stdout.

Done by Louie. The code now uses a simple print().

> I tried:
>
> $ ./python -m unittest test.test_builtin --list-tests
> doctest.DocTestCase.runTest
> doctest.DocTestCase.runTest
> ...

Using Louie's implementation, it "just" works!

haypo@selma$ ./python -m test test_builtin --list-cases|grep ^builtins
builtins.bin
builtins.float.as_integer_ratio
...

Even if these test names are not fully qualified, they are accept by --match/--matchfile!

./python -m test test_builtin --match=builtins.bin -v
...
0:00:00 load avg: 0.12 [1/1] test_builtin
bin (builtins)
Doctest: builtins.bin ... ok
...

So it will be possible to implement my bisect feature on these doctests as well!

I pushed Louie Lu's implementation: add a new --list-cases option to regrtest. Serhiy proposed a different CLI:

* --list=methods
* --list=files
* --list=classes

But I chose to push Louie's patch anyway since its patch is super simple and doesn't break anything: you can still use the existing --list-tests to list test *files*. I don't need --list=classes yet, it may be added later.

Serhiy: I plan to backport --matchfile and --list-cases features to 2.7, 3.5 and 3.6 branches, once Louie writes the unit test I requested him. Are you ok with that?

I really need to get my bisect tool on all branches, because more and more often I have to bisect refernece leaks, tests leaking resources (files, leaking memory, etc.), etc. --matchfile and --list-cases are the key features to implement such bisection.
msg296176 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 10:29
Ah something else: doctests are not fully supported.

./python -m test test_extcall --list-cases

This command has not output, whereas Lib/test/test_extcall.py contains doctests.

test_builtins works because it uses:

def load_tests(loader, tests, pattern):
    from doctest import DocTestSuite
    tests.addTest(DocTestSuite(builtins))
    return tests

whereas test_extcall uses:

def test_main():
    support.run_doctest(sys.modules[__name__], True)

I see two options:

* Replace support.run_doctest() with doctest.DocTestSuite() in all tests
* Enhance --list-cases to discover doctests. support.run_doctest() calls doctest.testmod(), but doctest.testmod() has no API to list tests. testmod() lists tests and directly runs them. Maybe --list-cases can reuse doctest.DocTestSuite(), I don't know.

I chose to merge Louie Lu's PR anyway, since I prefer to move step by step. Supporting test_extcall can be done later.
msg296177 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 11:15
New changeset b0c58d3c569d1b9060248e665857b5df94817340 by Victor Stinner (Louie Lu) in branch 'master':
bpo-30523: Add --list-cases unittest (#2243)
https://github.com/python/cpython/commit/b0c58d3c569d1b9060248e665857b5df94817340
msg296180 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 11:40
I created bpo-30683: Enhance doctest support in regrtest --list-cases.
msg296189 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 12:39
New changeset a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb by Victor Stinner in branch '3.6':
Synchronize libregrtest from master to 3.6 (#2244)
https://github.com/python/cpython/commit/a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb
msg296201 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 15:30
New changeset 24c2c20873dc800c99d1dabf26419b40cadfe627 by Victor Stinner in branch '2.7':
bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (#2249)
https://github.com/python/cpython/commit/24c2c20873dc800c99d1dabf26419b40cadfe627
msg296207 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 15:45
On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

* test_multibytecodec_support 
* test_robotparser
* test_xpickle

regrtest of Python 2.7 uses the test_main() function of each test file, if the function is defined. Whereas --list-cases is more written as a raw discovery function listing all cases which inherit from unittest.TestCase. These 3 files have base test cases which are "abstract" or configured below.

Options:

* Modify --list-cases to tolerate failures and report them at exit
* Modify these 3 tests to make them more "test discovery"-friendly
* Use test_main() function if available but modify its behaviour to only list test cases, instead of running them

I would prefer to limit changes in Python 2.7 branch, so IMHO the second option is the best: modify the 3 tests.
msg296208 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-16 16:04
New changeset 46e299c1536da541b9dbf76c0cb909bf8c79f589 by Victor Stinner in branch '3.5':
[3.5] bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (#2250)
https://github.com/python/cpython/commit/46e299c1536da541b9dbf76c0cb909bf8c79f589
msg296863 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 09:44
> On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

I created bpo-30759: "[2.7] Fix python2 -m test --list-cases test_multibytecodec_support test_robotparser".
msg296864 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 09:47
regrtest now supports --list-cases and --matchfile features in 2.7, 3.5, 3.6 and master (future 3.7) branches, so I now close the issue.

See bpo-30759 for the pending bug in Python 2.7. IMHO this bug is not a blocking feature, so it's ok to this close this issue.

Thanks Serhiy Storchaka and Julien Palard for the feedback and reviews, thanks Louie Lu for the actual implementation, much simpler than what I expected ;-)

These features already helped me a lot of identified reference leaks. So thank you all ;-)
msg296884 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-26 12:18
New changeset ace56d583664f855d89d1219ece7c21c2fddcf30 by Victor Stinner in branch 'master':
bpo-30523: regrtest --list-cases --match (#2401)
https://github.com/python/cpython/commit/ace56d583664f855d89d1219ece7c21c2fddcf30
msg297035 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:04
New changeset 35d2ca2b94a6ff29e763ddb7727166f0592edfa2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2441)
https://github.com/python/cpython/commit/35d2ca2b94a6ff29e763ddb7727166f0592edfa2
msg297042 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:35
New changeset de1850bb03f8225cbff85f437b6e972bf9b68c2a by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2442)
https://github.com/python/cpython/commit/de1850bb03f8225cbff85f437b6e972bf9b68c2a
msg297045 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-27 14:56
New changeset fea98bfcff6ccf9142daa97677fe86c1fdf8e63e by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2444)
https://github.com/python/cpython/commit/fea98bfcff6ccf9142daa97677fe86c1fdf8e63e
History
Date User Action Args
2017-06-27 14:56:45vstinnersetmessages: + msg297045
2017-06-27 14:49:29vstinnersetpull_requests: + pull_request2498
2017-06-27 14:35:20vstinnersetmessages: + msg297042
2017-06-27 14:15:35vstinnersetpull_requests: + pull_request2494
2017-06-27 14:04:18vstinnersetmessages: + msg297035
2017-06-27 13:50:38vstinnersetpull_requests: + pull_request2491
2017-06-26 12:18:53vstinnersetmessages: + msg296884
2017-06-26 10:58:38vstinnersetpull_requests: + pull_request2448
2017-06-26 09:47:46vstinnersetstatus: open -> closed
versions: + Python 2.7, Python 3.5, Python 3.6
messages: + msg296864

resolution: fixed
stage: resolved
2017-06-26 09:44:39vstinnersetmessages: + msg296863
2017-06-16 16:04:40vstinnersetmessages: + msg296208
2017-06-16 15:45:30vstinnersetmessages: + msg296207
2017-06-16 15:35:16vstinnersettitle: unittest: add --list-tests option to only display the list of test names, don't run tests -> regrtest: add --list-cases option to only display test case identifiers
2017-06-16 15:30:05vstinnersetmessages: + msg296201
2017-06-16 15:29:09vstinnersetpull_requests: + pull_request2301
2017-06-16 15:11:08vstinnersetpull_requests: + pull_request2299
2017-06-16 12:39:11vstinnersetmessages: + msg296189
2017-06-16 11:40:52vstinnersetmessages: + msg296180
2017-06-16 11:35:58vstinnersetpull_requests: + pull_request2288
2017-06-16 11:15:30vstinnersetmessages: + msg296177
2017-06-16 10:38:03louielusetpull_requests: + pull_request2287
2017-06-16 10:29:24vstinnersetmessages: + msg296176
2017-06-16 10:24:02vstinnersetmessages: + msg296175
2017-06-16 09:36:21vstinnersetmessages: + msg296170
2017-06-16 03:49:31louielusetnosy: + louielu
messages: + msg296158
2017-06-16 03:45:05louielusetpull_requests: + pull_request2283
2017-06-09 08:43:30serhiy.storchakasetmessages: + msg295500
2017-06-05 05:07:40serhiy.storchakasetmessages: + msg295166
2017-06-04 22:13:16vstinnersetmessages: + msg295146
2017-06-03 16:42:05mdksetmessages: + msg295087
2017-06-03 15:50:03serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg295084
2017-05-31 12:28:25vstinnersetmessages: + msg294840
2017-05-31 09:56:46mdksetnosy: + mdk
messages: + msg294830
2017-05-31 09:30:12vstinnersetmessages: + msg294829
2017-05-31 09:28:12vstinnersetmessages: + msg294828
2017-05-31 09:26:05vstinnersetpull_requests: + pull_request1961
2017-05-31 09:22:37vstinnercreate