Attached pull requests adds a --list-tests option to only display the list of test names, don't run tests.

It's a first step to implement the issue bpo-29512: "regrtest refleak: implement bisection feature".

Maybe the feature already exists but I failed to find it. Since I'm suprised that nobody implements that before (in unittest), I wait for a first feedback before starting to document the new feature and to write an unit test for it.

support.run_unittest() doesn't use unittest.main() and so don't support the new --list-tests option.

I wrote this patch because I'm trying to find which test of test_threading leaks a reference:

./python -m test -R 3:3 test_threading

Usually, I modify Lib/test/test_threading.py to remove tests one by one. I "bisect" the test manually. But this process is slow and painful, and it seems like we have many tests which leak references!

If it can be of any help, pytest have a "pytest --collect-only", it may make sense to use the same option name for consistency. I also though about "--dry-run" but I still prefer --list-tests or --collect-only.

I used testr which has a --list-tests option:
http://testrepository.readthedocs.io/en/latest/MANUAL.html#listing-tests

How to use this feature? "./python -m unittest --list-tests" produces nothing.

It works in "discover" mode, I tested:

./python -m unittest discover ./Lib/unittest/test/ --list-tests

and it worked.

So maybe the --list-tests should be moved to _getDiscoveryArgParser?

Also spotted that tests are printed on stderr, typically when the users asks for something, it has to be printed to stdout. Typically usages are written on stderr in case of error, but on stdout when explicitly asked via --help, see:

$ ./python -m unittest discover ./Lib/unittest/test/ --tabayo | wc
usage: python -m unittest discover [-h] [-v] [-q] [--locals] [-f] [-c] [-b]
                                   [-s START] [-p PATTERN] [-t TOP]
                                   [--list-tests]
python -m unittest discover: error: unrecognized arguments: --tabayo
      0       0       0
$ ./python -m unittest discover ./Lib/unittest/test/ --help | wc
     23     129    1110

I used ./python -m test Lib/test/test_threading.py --list-tests.

I would like to be able to use this feature in regrtest too:

./python -m test test_threading --list-tests

Sadly, this option already exists and only return test files, not test
methods.

I concur with Julien, the result should be printed to stdout.

I tried:

$ ./python -m unittest test.test_builtin --list-tests
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
test.test_builtin.BuiltinTest.test_abs
test.test_builtin.BuiltinTest.test_all
...

$ ./python -m test.test_builtin --list-tests
BuiltinTest.test_abs
BuiltinTest.test_all
BuiltinTest.test_any
...
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest

All these doctest.DocTestCase.runTest look useless.

But "./python -m unittest test.test_json --list-tests" and "./python -m test.test_json --list-tests" give the same output.

"./python -m test.test_doctest --list-tests" and "./python -m test.test_tk --list-tests" run tests instead of listing them.

The main problem is doctests. Numerous "doctest.DocTestCase.runTest" look useless and confusing.

Running tests instead of listing them is bad.

It is not clear why in one cases --list-tests outputs full qualified test names, while in other cases it outputs only class and method names.

Serhiy and Victor suggest moving this option to regrtest. open a new PR 2238 addressed this.

Also, because moving into regrtest, some test that used requires will need to pass option when using it. e.g.:

  $ ./python -m test test.test_curses --list-cases -ucurses

otherwise, it will just passout and do nothing (maybe it should warn user?)

New changeset a49c935 by Victor Stinner (mlouielu) in branch 'master':
bpo-30523: regrtest: Add --list-cases option (bpo-2238)
a49c935

Yeah, Louie fixed most issues!

I concur with Julien, the result should be printed to stdout.

Done by Louie. The code now uses a simple print().

I tried:

$ ./python -m unittest test.test_builtin --list-tests
doctest.DocTestCase.runTest
doctest.DocTestCase.runTest
...

Using Louie's implementation, it "just" works!

haypo@selma$ ./python -m test test_builtin --list-cases|grep ^builtins
builtins.bin
builtins.float.as_integer_ratio
...

Even if these test names are not fully qualified, they are accept by --match/--matchfile!

./python -m test test_builtin --match=builtins.bin -v
...
0:00:00 load avg: 0.12 [1/1] test_builtin
bin (builtins)
Doctest: builtins.bin ... ok
...

So it will be possible to implement my bisect feature on these doctests as well!

I pushed Louie Lu's implementation: add a new --list-cases option to regrtest. Serhiy proposed a different CLI:

• --list=methods
• --list=files
• --list=classes

But I chose to push Louie's patch anyway since its patch is super simple and doesn't break anything: you can still use the existing --list-tests to list test *files*. I don't need --list=classes yet, it may be added later.

Serhiy: I plan to backport --matchfile and --list-cases features to 2.7, 3.5 and 3.6 branches, once Louie writes the unit test I requested him. Are you ok with that?

I really need to get my bisect tool on all branches, because more and more often I have to bisect refernece leaks, tests leaking resources (files, leaking memory, etc.), etc. --matchfile and --list-cases are the key features to implement such bisection.

Ah something else: doctests are not fully supported.

./python -m test test_extcall --list-cases

This command has not output, whereas Lib/test/test_extcall.py contains doctests.

test_builtins works because it uses:

def load_tests(loader, tests, pattern):
    from doctest import DocTestSuite
    tests.addTest(DocTestSuite(builtins))
    return tests

whereas test_extcall uses:

def test_main():
    support.run_doctest(sys.modules[__name__], True)

I see two options:

• Replace support.run_doctest() with doctest.DocTestSuite() in all tests
• Enhance --list-cases to discover doctests. support.run_doctest() calls doctest.testmod(), but doctest.testmod() has no API to list tests. testmod() lists tests and directly runs them. Maybe --list-cases can reuse doctest.DocTestSuite(), I don't know.

I chose to merge Louie Lu's PR anyway, since I prefer to move step by step. Supporting test_extcall can be done later.

New changeset b0c58d3 by Victor Stinner (Louie Lu) in branch 'master':
bpo-30523: Add --list-cases unittest (bpo-2243)
b0c58d3

I created bpo-30683: Enhance doctest support in regrtest --list-cases.

New changeset a0ccc54 by Victor Stinner in branch '3.6':
Synchronize libregrtest from master to 3.6 (bpo-2244)
a0ccc54

New changeset 24c2c20 by Victor Stinner in branch '2.7':
bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (bpo-2249)
24c2c20

On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

• test_multibytecodec_support
• test_robotparser
• test_xpickle

regrtest of Python 2.7 uses the test_main() function of each test file, if the function is defined. Whereas --list-cases is more written as a raw discovery function listing all cases which inherit from unittest.TestCase. These 3 files have base test cases which are "abstract" or configured below.

Options:

• Modify --list-cases to tolerate failures and report them at exit
• Modify these 3 tests to make them more "test discovery"-friendly
• Use test_main() function if available but modify its behaviour to only list test cases, instead of running them

I would prefer to limit changes in Python 2.7 branch, so IMHO the second option is the best: modify the 3 tests.

New changeset 46e299c by Victor Stinner in branch '3.5':
[3.5] bpo-30540, bpo-30523: Add --matchfile and --list-cases options to regrtest (bpo-2250)
46e299c

On Python 2.7, listing *all* test cases with "./python -m test --list-cases" fails on the following test files:

I created bpo-30759: "[2.7] Fix python2 -m test --list-cases test_multibytecodec_support test_robotparser".

regrtest now supports --list-cases and --matchfile features in 2.7, 3.5, 3.6 and master (future 3.7) branches, so I now close the issue.

See bpo-30759 for the pending bug in Python 2.7. IMHO this bug is not a blocking feature, so it's ok to this close this issue.

Thanks Serhiy Storchaka and Julien Palard for the feedback and reviews, thanks Louie Lu for the actual implementation, much simpler than what I expected ;-)

These features already helped me a lot of identified reference leaks. So thank you all ;-)

New changeset ace56d5 by Victor Stinner in branch 'master':
bpo-30523: regrtest --list-cases --match (bpo-2401)
ace56d5

New changeset 35d2ca2 by Victor Stinner in branch '3.6':
[3.6] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2441)
35d2ca2

New changeset de1850b by Victor Stinner in branch '3.5':
[3.5] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2442)
de1850b

New changeset fea98bf by Victor Stinner in branch '2.7':
[2.7] bpo-30523, bpo-30764, bpo-30776: Sync regrtest from master (bpo-2444)
fea98bf