Currently regrtest supports two ways of collecting and running tests in module.

1. If the module contains the "test_main" function, regrtest just calls it. This function usually calls run_unittest() with a list of test classes, composed manually, it can also run doctests with run_doctest(), generate list of test classes dynamically, and run some code before and after running tests.

2. Otherwise regrtest uses unittest for for collecting tests.

The disadvantage of the former way is that new test classes should be added manually to the list. If you forget to do this, new tests will not be run. See for example issue45185 and issue45187. Not runned tests can hide bugs.

So it would be better to eliminate human factor and detect tests automatically.

PR 28405 contains simple changes for 60 test files. It does not cover files which use complex code for generating list of test classes and running doctests (because there are subtle differences between running doctests with run_doctest() and via DocTestSuite). They are left for the following PRs.

See also bpo-16748

Thank you Zachary, I almost forgot about that issue.

The goal of this issue is to make regrtests using test discovery instead of manually composed lists of test classes. Of course it requires that tests should be discoverable. And most of them already are, thanks to issue16748 and similar.

New changeset 40348acc180580371d25f75f46b27048e35f2435 by Serhiy Storchaka in branch 'main':
bpo-45229: Remove test_main in many tests (GH-28405)
https://github.com/python/cpython/commit/40348acc180580371d25f75f46b27048e35f2435

New changeset a6241773869b80f2b1cac63f26ba646aa808a8db by Serhiy Storchaka in branch 'main':
bpo-45229: Fix setUpModule in test_ssl (GH-28454)
https://github.com/python/cpython/commit/a6241773869b80f2b1cac63f26ba646aa808a8db

New changeset bedce3538cca3469ac3efc614ef062367cbb2ff1 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28455)
https://github.com/python/cpython/commit/bedce3538cca3469ac3efc614ef062367cbb2ff1

New changeset 5822ab672a1d26ff1837103c1ed8e4c3c2a42b87 by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28456)
https://github.com/python/cpython/commit/5822ab672a1d26ff1837103c1ed8e4c3c2a42b87

New changeset a856364cc920d8b16750fd1fadc902efb509754c by Serhiy Storchaka in branch 'main':
bpo-45229: Use doctest.DocTestSuite instead of run_doctest (GH-28468)
https://github.com/python/cpython/commit/a856364cc920d8b16750fd1fadc902efb509754c

Thank you for doing this.  When I started setting up the IDLE test suite 8 years ago, I was a bit confused about the mixture of old ways (test_main) and new ways (unittest.main and possibly load_tests) in the test suite.  Fortunately, Nick C. told me which was which.  Updating the test suite should make it better as an example for new unittest users.

New changeset e6ba992288fdbe71aa808cfb6955f6f99da7e349 by Serhiy Storchaka in branch 'main':
bpo-45229: Make pickle tests discoverable (GH-28467)
https://github.com/python/cpython/commit/e6ba992288fdbe71aa808cfb6955f6f99da7e349

New changeset 74c6acc76fa00f1b80e96c8d4608ba1487387949 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28478)
https://github.com/python/cpython/commit/74c6acc76fa00f1b80e96c8d4608ba1487387949

New changeset 04dea46dde25d58f242171181805f0e71a042963 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28479)
https://github.com/python/cpython/commit/04dea46dde25d58f242171181805f0e71a042963

New changeset d441437ee71ae174c008c23308b749b91020ba77 by Serhiy Storchaka in branch 'main':
bpo-45229: Make datetime tests discoverable (GH-28615)
https://github.com/python/cpython/commit/d441437ee71ae174c008c23308b749b91020ba77

Unfortunately, this PR28615 seems to have broken a bunch of buildbots. For instance:

https://buildbot.python.org/all/#/builders/75/builds/172/steps/5/logs/stdio

```
test_divide_and_round (test.datetimetester.TestModule_Pure) ... ok
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=1)
test_divide_and_round (test.datetimetester.TestModule_Pure) ... ERROR
.test test_datetime failed
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
======================================================================
ERROR: test_divide_and_round (test.datetimetester.TestModule_Pure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'datetime' has no attribute '_divide_and_round'
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (errors=1, skipped=1)
1 test failed again:
    test_datetime
== Tests result: FAILURE then FAILURE ==
```

Seems this is not a new bug:

$ ./python -m test test_datetime test_datetime -m test_divide_and_round
0:00:00 load avg: 2.36 Run tests sequentially
0:00:00 load avg: 2.36 [1/2] test_datetime
0:00:00 load avg: 2.36 [2/2] test_datetime
test test_datetime failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.9/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
AttributeError: module 'datetime' has no attribute '_divide_and_round'

test_datetime failed (1 error)

== Tests result: FAILURE ==


But for some causes it was not reproduced with refleak tests.

> Seems this is not a new bug:

I agree, is unfortunate. But sadly is breaking all buildbots and therefore is hiding other issues and making the 'test-with-buildbots' label not useful. I would prefer to revert until we have a fix (and not land the backports until then).

See issue40058 and issue40173.

New changeset b07fddd527efe67174ce6b0fdbe8dac390b16e4e by Pablo Galindo Salgado in branch 'main':
Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)
https://github.com/python/cpython/commit/b07fddd527efe67174ce6b0fdbe8dac390b16e4e

New changeset 993a130d3abe7684dc9c999874b4dd1d8ea55a2a by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)
https://github.com/python/cpython/commit/993a130d3abe7684dc9c999874b4dd1d8ea55a2a

New changeset 2cf76cf4ccd177b8d6d2bf21b5462258ae87522d by Łukasz Langa in branch '3.9':
Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)
https://github.com/python/cpython/commit/2cf76cf4ccd177b8d6d2bf21b5462258ae87522d

New changeset ef7c7294e8404d844c1add892a8f6684e6cf4f31 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make datetime tests discoverable (GH-28615)
https://github.com/python/cpython/commit/ef7c7294e8404d844c1add892a8f6684e6cf4f31

New changeset 2f205920127bd93eebed044cb1b61834764478ba by Serhiy Storchaka in branch 'main':
Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)" (GH-28667)
https://github.com/python/cpython/commit/2f205920127bd93eebed044cb1b61834764478ba

New changeset e9d5cdda1ffa369550a634a3ec220db93433e0f4 by Serhiy Storchaka in branch '3.9':
Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)" (GH-28666)
https://github.com/python/cpython/commit/e9d5cdda1ffa369550a634a3ec220db93433e0f4

New changeset f59ed3c310a7ceebf2a56a84ea969a7f75d95b64 by Serhiy Storchaka in branch 'main':
bpo-45229: Make tkinter tests discoverable (GH-28637)
https://github.com/python/cpython/commit/f59ed3c310a7ceebf2a56a84ea969a7f75d95b64

New changeset 151234f5da4b1922e3d91233a6eddda92d7269b0 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28927)
https://github.com/python/cpython/commit/151234f5da4b1922e3d91233a6eddda92d7269b0

New changeset 5d747130933a25a3984f78ae32d2e74a0d2dc5d1 by Łukasz Langa in branch '3.9':
[3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28929)
https://github.com/python/cpython/commit/5d747130933a25a3984f78ae32d2e74a0d2dc5d1

New changeset b3f0ceae919c1627094ff628c87184684a5cedd6 by Serhiy Storchaka in branch 'main':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/b3f0ceae919c1627094ff628c87184684a5cedd6

New changeset 1dbf9c86b25463b9bc695e434ac034a7e313ba01 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/1dbf9c86b25463b9bc695e434ac034a7e313ba01

New changeset 65c1db794e0484e18142658c691141387c1dc2e4 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/65c1db794e0484e18142658c691141387c1dc2e4

New changeset 8d6740f489fca67a44de165d29d9e0ad86285779 by Serhiy Storchaka in branch 'main':
bpo-45229: Make doctest tests discoverable (GH-28986)
https://github.com/python/cpython/commit/8d6740f489fca67a44de165d29d9e0ad86285779

New changeset 65de808811f93793599209f74bb1bab3ad399b17 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29095)
https://github.com/python/cpython/commit/65de808811f93793599209f74bb1bab3ad399b17

New changeset 919268316582c6ac47960c2e5dd2ee1682371494 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29096)
https://github.com/python/cpython/commit/919268316582c6ac47960c2e5dd2ee1682371494