classification
Title: Always use unittest for collecting tests in regrtests
Type: enhancement Stage: patch review
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: 40058 40173 45185 45187 45500 45502 Superseder:
Assigned To: serhiy.storchaka Nosy List: jerrylikerice1s, lukasz.langa, lukasz.langa, miss-islington, pablogsal, serhiy.storchaka, terry.reedy, zach.ware
Priority: normal Keywords: patch

Created on 2021-09-17 07:31 by serhiy.storchaka, last changed 2021-10-20 16:52 by lukasz.langa.

Pull Requests
URL Status Linked Edit
PR 28405 merged serhiy.storchaka, 2021-09-17 07:40
PR 28454 merged serhiy.storchaka, 2021-09-19 12:38
PR 28455 merged serhiy.storchaka, 2021-09-19 12:42
PR 28456 merged serhiy.storchaka, 2021-09-19 13:19
PR 28467 merged serhiy.storchaka, 2021-09-20 07:42
PR 28468 merged serhiy.storchaka, 2021-09-20 07:45
PR 28478 merged miss-islington, 2021-09-20 15:21
PR 28479 merged miss-islington, 2021-09-20 15:21
PR 28615 merged serhiy.storchaka, 2021-09-29 10:22
PR 28637 merged serhiy.storchaka, 2021-09-29 20:24
PR 28639 closed miss-islington, 2021-09-29 20:28
PR 28645 merged serhiy.storchaka, 2021-09-30 07:26
PR 28650 merged pablogsal, 2021-09-30 12:21
PR 28660 merged lukasz.langa, 2021-09-30 19:40
PR 28666 merged serhiy.storchaka, 2021-10-01 07:11
PR 28667 merged serhiy.storchaka, 2021-10-01 07:11
PR 28668 merged miss-islington, 2021-10-01 07:14
PR 28927 merged miss-islington, 2021-10-13 16:12
PR 28929 merged lukasz.langa, 2021-10-13 16:26
PR 28986 merged serhiy.storchaka, 2021-10-15 20:35
PR 29004 merged serhiy.storchaka, 2021-10-17 10:26
PR 29006 merged miss-islington, 2021-10-17 12:06
PR 29007 merged miss-islington, 2021-10-17 12:06
PR 29095 merged miss-islington, 2021-10-20 14:36
PR 29096 merged miss-islington, 2021-10-20 14:36
Messages (33)
msg402006 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-17 07:31
Currently regrtest supports two ways of collecting and running tests in module.

1. If the module contains the "test_main" function, regrtest just calls it. This function usually calls run_unittest() with a list of test classes, composed manually, it can also run doctests with run_doctest(), generate list of test classes dynamically, and run some code before and after running tests.

2. Otherwise regrtest uses unittest for for collecting tests.

The disadvantage of the former way is that new test classes should be added manually to the list. If you forget to do this, new tests will not be run. See for example issue45185 and issue45187. Not runned tests can hide bugs.

So it would be better to eliminate human factor and detect tests automatically.
msg402009 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-17 07:44
PR 28405 contains simple changes for 60 test files. It does not cover files which use complex code for generating list of test classes and running doctests (because there are subtle differences between running doctests with run_doctest() and via DocTestSuite). They are left for the following PRs.
msg402053 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2021-09-17 13:45
See also bpo-16748
msg402076 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-17 19:15
Thank you Zachary, I almost forgot about that issue.

The goal of this issue is to make regrtests using test discovery instead of manually composed lists of test classes. Of course it requires that tests should be discoverable. And most of them already are, thanks to issue16748 and similar.
msg402154 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-19 12:27
New changeset 40348acc180580371d25f75f46b27048e35f2435 by Serhiy Storchaka in branch 'main':
bpo-45229: Remove test_main in many tests (GH-28405)
https://github.com/python/cpython/commit/40348acc180580371d25f75f46b27048e35f2435
msg402156 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-19 13:18
New changeset a6241773869b80f2b1cac63f26ba646aa808a8db by Serhiy Storchaka in branch 'main':
bpo-45229: Fix setUpModule in test_ssl (GH-28454)
https://github.com/python/cpython/commit/a6241773869b80f2b1cac63f26ba646aa808a8db
msg402168 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-19 19:36
New changeset bedce3538cca3469ac3efc614ef062367cbb2ff1 by Serhiy Storchaka in branch '3.10':
[3.10] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28455)
https://github.com/python/cpython/commit/bedce3538cca3469ac3efc614ef062367cbb2ff1
msg402195 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-20 06:34
New changeset 5822ab672a1d26ff1837103c1ed8e4c3c2a42b87 by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45229: Remove test_main in many tests (GH-28405) (GH-28456)
https://github.com/python/cpython/commit/5822ab672a1d26ff1837103c1ed8e4c3c2a42b87
msg402214 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-20 08:37
New changeset a856364cc920d8b16750fd1fadc902efb509754c by Serhiy Storchaka in branch 'main':
bpo-45229: Use doctest.DocTestSuite instead of run_doctest (GH-28468)
https://github.com/python/cpython/commit/a856364cc920d8b16750fd1fadc902efb509754c
msg402228 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-09-20 13:00
Thank you for doing this.  When I started setting up the IDLE test suite 8 years ago, I was a bit confused about the mixture of old ways (test_main) and new ways (unittest.main and possibly load_tests) in the test suite.  Fortunately, Nick C. told me which was which.  Updating the test suite should make it better as an example for new unittest users.
msg402245 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-20 15:21
New changeset e6ba992288fdbe71aa808cfb6955f6f99da7e349 by Serhiy Storchaka in branch 'main':
bpo-45229: Make pickle tests discoverable (GH-28467)
https://github.com/python/cpython/commit/e6ba992288fdbe71aa808cfb6955f6f99da7e349
msg402254 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-20 16:19
New changeset 74c6acc76fa00f1b80e96c8d4608ba1487387949 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28478)
https://github.com/python/cpython/commit/74c6acc76fa00f1b80e96c8d4608ba1487387949
msg402255 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-20 16:19
New changeset 04dea46dde25d58f242171181805f0e71a042963 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make pickle tests discoverable (GH-28467) (GH-28479)
https://github.com/python/cpython/commit/04dea46dde25d58f242171181805f0e71a042963
msg402903 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-29 20:28
New changeset d441437ee71ae174c008c23308b749b91020ba77 by Serhiy Storchaka in branch 'main':
bpo-45229: Make datetime tests discoverable (GH-28615)
https://github.com/python/cpython/commit/d441437ee71ae174c008c23308b749b91020ba77
msg402945 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-30 10:39
Unfortunately, this PR28615 seems to have broken a bunch of buildbots. For instance:

https://buildbot.python.org/all/#/builders/75/builds/172/steps/5/logs/stdio

```
test_divide_and_round (test.datetimetester.TestModule_Pure) ... ok
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK (skipped=1)
test_divide_and_round (test.datetimetester.TestModule_Pure) ... ERROR
.test test_datetime failed
test_divide_and_round (test.datetimetester.TestModule_Fast) ... skipped 'Only run for Pure Python implementation'
======================================================================
ERROR: test_divide_and_round (test.datetimetester.TestModule_Pure)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z.refleak/build/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'datetime' has no attribute '_divide_and_round'
----------------------------------------------------------------------
Ran 2 tests in 0.003s
FAILED (errors=1, skipped=1)
1 test failed again:
    test_datetime
== Tests result: FAILURE then FAILURE ==
```
msg402952 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 12:08
Seems this is not a new bug:

$ ./python -m test test_datetime test_datetime -m test_divide_and_round
0:00:00 load avg: 2.36 Run tests sequentially
0:00:00 load avg: 2.36 [1/2] test_datetime
0:00:00 load avg: 2.36 [2/2] test_datetime
test test_datetime failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.9/Lib/test/datetimetester.py", line 88, in test_divide_and_round
    dar = datetime_module._divide_and_round
AttributeError: module 'datetime' has no attribute '_divide_and_round'

test_datetime failed (1 error)

== Tests result: FAILURE ==


But for some causes it was not reproduced with refleak tests.
msg402953 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-30 12:21
> Seems this is not a new bug:

I agree, is unfortunate. But sadly is breaking all buildbots and therefore is hiding other issues and making the 'test-with-buildbots' label not useful. I would prefer to revert until we have a fix (and not land the backports until then).
msg402956 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 13:18
See issue40058 and issue40173.
msg402959 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2021-09-30 13:46
New changeset b07fddd527efe67174ce6b0fdbe8dac390b16e4e by Pablo Galindo Salgado in branch 'main':
Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)
https://github.com/python/cpython/commit/b07fddd527efe67174ce6b0fdbe8dac390b16e4e
msg402974 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-30 16:58
New changeset 993a130d3abe7684dc9c999874b4dd1d8ea55a2a by Serhiy Storchaka in branch '3.9':
[3.9] bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)
https://github.com/python/cpython/commit/993a130d3abe7684dc9c999874b4dd1d8ea55a2a
msg402982 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-09-30 20:23
New changeset 2cf76cf4ccd177b8d6d2bf21b5462258ae87522d by Łukasz Langa in branch '3.9':
Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)
https://github.com/python/cpython/commit/2cf76cf4ccd177b8d6d2bf21b5462258ae87522d
msg402996 - (view) Author: miss-islington (miss-islington) Date: 2021-10-01 07:36
New changeset ef7c7294e8404d844c1add892a8f6684e6cf4f31 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make datetime tests discoverable (GH-28615)
https://github.com/python/cpython/commit/ef7c7294e8404d844c1add892a8f6684e6cf4f31
msg402999 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-10-01 07:58
New changeset 2f205920127bd93eebed044cb1b61834764478ba by Serhiy Storchaka in branch 'main':
Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)" (GH-28667)
https://github.com/python/cpython/commit/2f205920127bd93eebed044cb1b61834764478ba
msg403000 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-10-01 07:59
New changeset e9d5cdda1ffa369550a634a3ec220db93433e0f4 by Serhiy Storchaka in branch '3.9':
Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)" (GH-28666)
https://github.com/python/cpython/commit/e9d5cdda1ffa369550a634a3ec220db93433e0f4
msg403846 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-13 16:12
New changeset f59ed3c310a7ceebf2a56a84ea969a7f75d95b64 by Serhiy Storchaka in branch 'main':
bpo-45229: Make tkinter tests discoverable (GH-28637)
https://github.com/python/cpython/commit/f59ed3c310a7ceebf2a56a84ea969a7f75d95b64
msg403853 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-13 16:59
New changeset 151234f5da4b1922e3d91233a6eddda92d7269b0 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28927)
https://github.com/python/cpython/commit/151234f5da4b1922e3d91233a6eddda92d7269b0
msg403855 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-13 17:08
New changeset 5d747130933a25a3984f78ae32d2e74a0d2dc5d1 by Łukasz Langa in branch '3.9':
[3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28929)
https://github.com/python/cpython/commit/5d747130933a25a3984f78ae32d2e74a0d2dc5d1
msg404137 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-10-17 12:06
New changeset b3f0ceae919c1627094ff628c87184684a5cedd6 by Serhiy Storchaka in branch 'main':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/b3f0ceae919c1627094ff628c87184684a5cedd6
msg404142 - (view) Author: miss-islington (miss-islington) Date: 2021-10-17 16:47
New changeset 1dbf9c86b25463b9bc695e434ac034a7e313ba01 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/1dbf9c86b25463b9bc695e434ac034a7e313ba01
msg404143 - (view) Author: miss-islington (miss-islington) Date: 2021-10-17 16:47
New changeset 65c1db794e0484e18142658c691141387c1dc2e4 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make test_http_cookiejar discoverable (GH-29004)
https://github.com/python/cpython/commit/65c1db794e0484e18142658c691141387c1dc2e4
msg404466 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-20 14:36
New changeset 8d6740f489fca67a44de165d29d9e0ad86285779 by Serhiy Storchaka in branch 'main':
bpo-45229: Make doctest tests discoverable (GH-28986)
https://github.com/python/cpython/commit/8d6740f489fca67a44de165d29d9e0ad86285779
msg404495 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-20 16:52
New changeset 65de808811f93793599209f74bb1bab3ad399b17 by Miss Islington (bot) in branch '3.10':
bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29095)
https://github.com/python/cpython/commit/65de808811f93793599209f74bb1bab3ad399b17
msg404496 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-10-20 16:52
New changeset 919268316582c6ac47960c2e5dd2ee1682371494 by Miss Islington (bot) in branch '3.9':
bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29096)
https://github.com/python/cpython/commit/919268316582c6ac47960c2e5dd2ee1682371494
History
Date User Action Args
2021-10-20 16:52:45lukasz.langasetmessages: + msg404496
2021-10-20 16:52:31lukasz.langasetmessages: + msg404495
2021-10-20 14:36:42miss-islingtonsetpull_requests: + pull_request27368
2021-10-20 14:36:37miss-islingtonsetpull_requests: + pull_request27367
2021-10-20 14:36:37lukasz.langasetmessages: + msg404466
2021-10-17 16:47:07miss-islingtonsetmessages: + msg404143
2021-10-17 16:47:01miss-islingtonsetmessages: + msg404142
2021-10-17 12:06:53miss-islingtonsetpull_requests: + pull_request27288
2021-10-17 12:06:48miss-islingtonsetpull_requests: + pull_request27287
2021-10-17 12:06:48serhiy.storchakasetmessages: + msg404137
2021-10-17 10:26:38serhiy.storchakasetpull_requests: + pull_request27286
2021-10-17 10:21:57serhiy.storchakasetdependencies: + Rewrite test_dbm
2021-10-17 10:21:20serhiy.storchakasetdependencies: + Fix test_shelve and make it discoverable
2021-10-15 20:35:41serhiy.storchakasetpull_requests: + pull_request27272
2021-10-15 11:38:19jerrylikerice1ssetnosy: + jerrylikerice1s
2021-10-13 17:08:48lukasz.langasetmessages: + msg403855
2021-10-13 16:59:07lukasz.langasetmessages: + msg403853
2021-10-13 16:26:07lukasz.langasetpull_requests: + pull_request27218
2021-10-13 16:12:58miss-islingtonsetpull_requests: + pull_request27216
2021-10-13 16:12:52lukasz.langasetmessages: + msg403846
2021-10-01 07:59:03serhiy.storchakasetmessages: + msg403000
2021-10-01 07:58:03serhiy.storchakasetmessages: + msg402999
2021-10-01 07:36:33miss-islingtonsetmessages: + msg402996
2021-10-01 07:14:02miss-islingtonsetpull_requests: + pull_request27033
2021-10-01 07:11:52serhiy.storchakasetpull_requests: + pull_request27032
2021-10-01 07:11:23serhiy.storchakasetpull_requests: + pull_request27031
2021-09-30 20:23:45lukasz.langasetmessages: + msg402982
2021-09-30 19:40:19lukasz.langasetpull_requests: + pull_request27025
2021-09-30 16:58:38serhiy.storchakasetmessages: + msg402974
2021-09-30 13:46:33pablogsalsetmessages: + msg402959
2021-09-30 13:18:30serhiy.storchakasetdependencies: + Running test_datetime twice fails with: module 'datetime' has no attribute '_divide_and_round', test.support.import_fresh_module fails to correctly block submodules when fresh is specified
messages: + msg402956
2021-09-30 12:21:30pablogsalsetpull_requests: + pull_request27018
2021-09-30 12:21:05pablogsalsetmessages: + msg402953
2021-09-30 12:08:49serhiy.storchakasetmessages: + msg402952
2021-09-30 10:39:19pablogsalsetnosy: + pablogsal
messages: + msg402945
2021-09-30 07:26:43serhiy.storchakasetpull_requests: + pull_request27013
2021-09-29 20:28:05miss-islingtonsetpull_requests: + pull_request27008
2021-09-29 20:28:04serhiy.storchakasetmessages: + msg402903
2021-09-29 20:24:50serhiy.storchakasetpull_requests: + pull_request27006
2021-09-29 10:22:23serhiy.storchakasetpull_requests: + pull_request26986
2021-09-20 16:19:53lukasz.langasetnosy: + lukasz.langa
messages: + msg402255
2021-09-20 16:19:53lukasz.langasetnosy: + lukasz.langa
messages: + msg402254
2021-09-20 15:21:47miss-islingtonsetpull_requests: + pull_request26877
2021-09-20 15:21:42miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26876
2021-09-20 15:21:41serhiy.storchakasetmessages: + msg402245
2021-09-20 13:00:34terry.reedysetnosy: + terry.reedy
messages: + msg402228
2021-09-20 08:37:04serhiy.storchakasetmessages: + msg402214
2021-09-20 07:45:08serhiy.storchakasetpull_requests: + pull_request26867
2021-09-20 07:42:58serhiy.storchakasetpull_requests: + pull_request26866
2021-09-20 06:34:59serhiy.storchakasetmessages: + msg402195
2021-09-19 19:36:11serhiy.storchakasetmessages: + msg402168
2021-09-19 13:19:15serhiy.storchakasetpull_requests: + pull_request26858
2021-09-19 13:18:19serhiy.storchakasetmessages: + msg402156
2021-09-19 12:42:46serhiy.storchakasetpull_requests: + pull_request26857
2021-09-19 12:38:41serhiy.storchakasetpull_requests: + pull_request26856
2021-09-19 12:27:37serhiy.storchakasetmessages: + msg402154
2021-09-17 19:15:21serhiy.storchakasetmessages: + msg402076
2021-09-17 13:45:19zach.waresetnosy: + zach.ware
messages: + msg402053
2021-09-17 07:45:32serhiy.storchakasetdependencies: + test.test_ssl.TestEnumerations is not run, Some tests in test_socket are not run
2021-09-17 07:44:52serhiy.storchakasetmessages: + msg402009
2021-09-17 07:40:20serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request26817
2021-09-17 07:31:12serhiy.storchakacreate