Attached pull requests adds a --list-tests option to only display the list of test names, don't run tests.

It's a first step to implement the issue #29512: "regrtest refleak: implement bisection feature".

Maybe the feature already exists but I failed to find it. Since I'm suprised that nobody implements that before (in unittest), I wait for a first feedback before starting to document the new feature and to write an unit test for it.

support.run_unittest() doesn't use unittest.main() and so don't support the new --list-tests option.

I wrote this patch because I'm trying to find which test of test_threading leaks a reference:

./python -m test -R 3:3 test_threading

Usually, I modify Lib/test/test_threading.py to remove tests one by one. I "bisect" the test manually. But this process is slow and painful, and it seems like we have many tests which leak references!

If it can be of any help, pytest have a "pytest --collect-only", it may make sense to use the same option name for consistency. I also though about "--dry-run" but I still prefer --list-tests or --collect-only.

I used testr which has a --list-tests option:
http://testrepository.readthedocs.io/en/latest/MANUAL.html#listing-tests

How to use this feature? "./python -m unittest --list-tests" produces nothing.

It works in "discover" mode, I tested:

./python -m unittest discover ./Lib/unittest/test/ --list-tests

and it worked.

So maybe the --list-tests should be moved to _getDiscoveryArgParser?

Also spotted that tests are printed on stderr, typically when the users asks for something, it has to be printed to stdout. Typically usages are written on stderr in case of error, but on stdout when explicitly asked via --help, see:

$ ./python -m unittest discover ./Lib/unittest/test/ --tabayo | wc
usage: python -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b]
                                   [-s START] [-p PATTERN] [-t TOP]
                                   [--list-tests]
python -m unittest discover: error: unrecognized arguments: --tabayo
      0       0       0
$ ./python -m unittest discover ./Lib/unittest/test/ --help | wc
     23     129    1110

I used ./python -m test Lib/test/test_threading.py --list-tests.

I would like to be able to use this feature in regrtest too:

./python -m test test_threading --list-tests

Sadly, this option already exists and only return test files, not test
methods.

I concur with Julien, the result should be printed to stdout.

I tried:

$ ./python -m unittest test.test_builtin --list-tests
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
test.test_builtin.BuiltinTest.test_abs
test.test_builtin.BuiltinTest.test_all
...

$ ./python -m test.test_builtin --list-tests
BuiltinTest.test_abs
BuiltinTest.test_all
BuiltinTest.test_any
...
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest

All these doctest.DocTestCase.runTest look useless.

But "./python -m unittest test.test_json --list-tests" and "./python -m test.test_json --list-tests" give the same output.

"./python -m test.test_doctest --list-tests" and "./python -m test.test_tk --list-tests" run tests instead of listing them.

The main problem is doctests. Numerous "doctest.DocTestCase.runTest" look useless and confusing.

Running tests instead of listing them is bad.

It is not clear why in one cases --list-tests outputs full qualified test names, while in other cases it outputs only class and method names.

Serhiy and Victor suggest moving this option to regrtest. open a new PR 2238 addressed this.

Also, because moving into regrtest, some test that used `requires` will need to pass option when using it. e.g.:

  $ ./python -m test test.test_curses --list-cases -ucurses

otherwise, it will just passout and do nothing (maybe it should warn user?)

New changeset a49c935cfd6b4f8ea1f750888dd9260bbf5b9980 by Victor Stinner (mlouielu) in branch 'master':
bpo-30523: regrtest: Add --list-cases option (#2238)
https://github.com/python/cpython/commit/a49c935cfd6b4f8ea1f750888dd9260bbf5b9980

Yeah, Louie fixed most issues!

> I concur with Julien, the result should be printed to stdout.

Done by Louie. The code now uses a simple print().

> I tried:
>
> $ ./python -m unittest test.test_builtin --list-tests
> doctest.DocTestCase.runTest
> doctest.DocTestCase.runTest
> ...

Using Louie's implementation, it "just" works!

haypo@selma$ ./python -m test test_builtin --list-cases|grep ^builtins
builtins.bin
builtins.float.as_integer_ratio
...

Even if these test names are not fully qualified, they are accept by --match/--matchfile!

./python -m test test_builtin --match=builtins.bin -v
...
0:00:00 load avg: 0.12 [1/1] test_builtin
bin (builtins)
Doctest: builtins.bin ... ok
...

So it will be possible to implement my bisect feature on these doctests as well!

I pushed Louie Lu's implementation: add a new --list-cases option to regrtest. Serhiy proposed a different CLI:

* --list=methods
* --list=files
* --list=classes

But I chose to push Louie's patch anyway since its patch is super simple and doesn't break anything: you can still use the existing --list-tests to list test *files*. I don't need --list=classes yet, it may be added later.

Serhiy: I plan to backport --matchfile and --list-cases features to 2.7, 3.5 and 3.6 branches, once Louie writes the unit test I requested him. Are you ok with that?

I really need to get my bisect tool on all branches, because more and more often I have to bisect refernece leaks, tests leaking resources (files, leaking memory, etc.), etc. --matchfile and --list-cases are the key features to implement such bisection.

Ah something else: doctests are not fully supported.

./python -m test test_extcall --list-cases

This command has not output, whereas Lib/test/test_extcall.py contains doctests.

test_builtins works because it uses:

def load_tests(loader, tests, pattern):
    from doctest import DocTestSuite
    tests.addTest(DocTestSuite(builtins))
    return tests

whereas test_extcall uses:

def test_main():
    support.run_doctest(sys.modules[__name__], True)

I see two options:

* Replace support.run_doctest() with doctest.DocTestSuite() in all tests
* Enhance --list-cases to discover doctests. support.run_doctest() calls doctest.testmod(), but doctest.testmod() has no API to list tests. testmod() lists tests and directly runs them. Maybe --list-cases can reuse doctest.DocTestSuite(), I don't know.

I chose to merge Louie Lu's PR anyway, since I prefer to move step by step. Supporting test_extcall can be done later.

New changeset b0c58d3c569d1b9060248e665857b5df94817340 by Victor Stinner (Louie Lu) in branch 'master':
bpo-30523: Add --list-cases unittest (#2243)
https://github.com/python/cpython/commit/b0c58d3c569d1b9060248e665857b5df94817340

I created bpo-30683: Enhance doctest support in regrtest --list-cases.

New changeset a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb by Victor Stinner in branch '3.6':
Synchronize libregrtest from master to 3.6 (#2244)
https://github.com/python/cpython/commit/a0ccc54e6dffacf9e7c06f2a3e9056d2d35d21eb

New changeset 24c2c20873dc800c99d1dabf26419b40cadfe627 by Victor Stinner in branch '2.7':
bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (#2249)
https://github.com/python/cpython/commit/24c2c20873dc800c99d1dabf26419b40cadfe627

On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

* test_multibytecodec_support 
* test_robotparser
* test_xpickle

regrtest of Python 2.7 uses the test_main() function of each test file, if the function is defined. Whereas --list-cases is more written as a raw discovery function listing all cases which inherit from unittest.TestCase. These 3 files have base test cases which are "abstract" or configured below.

Options:

* Modify --list-cases to tolerate failures and report them at exit
* Modify these 3 tests to make them more "test discovery"-friendly
* Use test_main() function if available but modify its behaviour to only list test cases, instead of running them

I would prefer to limit changes in Python 2.7 branch, so IMHO the second option is the best: modify the 3 tests.

New changeset 46e299c1536da541b9dbf76c0cb909bf8c79f589 by Victor Stinner in branch '3.5':
[3.5] bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (#2250)
https://github.com/python/cpython/commit/46e299c1536da541b9dbf76c0cb909bf8c79f589

> On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

I created bpo-30759: "[2.7] Fix python2 -m test --list-cases test_multibytecodec_support test_robotparser".

regrtest now supports --list-cases and --matchfile features in 2.7, 3.5, 3.6 and master (future 3.7) branches, so I now close the issue.

See bpo-30759 for the pending bug in Python 2.7. IMHO this bug is not a blocking feature, so it's ok to this close this issue.

Thanks Serhiy Storchaka and Julien Palard for the feedback and reviews, thanks Louie Lu for the actual implementation, much simpler than what I expected ;-)

These features already helped me a lot of identified reference leaks. So thank you all ;-)

New changeset ace56d583664f855d89d1219ece7c21c2fddcf30 by Victor Stinner in branch 'master':
bpo-30523: regrtest --list-cases --match (#2401)
https://github.com/python/cpython/commit/ace56d583664f855d89d1219ece7c21c2fddcf30

New changeset 35d2ca2b94a6ff29e763ddb7727166f0592edfa2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2441)
https://github.com/python/cpython/commit/35d2ca2b94a6ff29e763ddb7727166f0592edfa2

New changeset de1850bb03f8225cbff85f437b6e972bf9b68c2a by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2442)
https://github.com/python/cpython/commit/de1850bb03f8225cbff85f437b6e972bf9b68c2a

New changeset fea98bfcff6ccf9142daa97677fe86c1fdf8e63e by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (#2444)
https://github.com/python/cpython/commit/fea98bfcff6ccf9142daa97677fe86c1fdf8e63e