Title: test.regrtest: add an option to run test.bisect_cmd on failed tests, use it on Refleaks buildbots
Type: Stage: resolved
Components: Tests Versions: Python 3.9
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: corona10, shihai1991, vstinner
Priority: normal Keywords: patch

Created on 2020-03-18 15:48 by vstinner, last changed 2020-05-18 15:51 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19246 merged vstinner, 2020-03-31 14:05
Messages (3)
msg364528 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-18 15:48
There are some tests which fail randomly in general, but fail in a deterministic way on some specific buildbot workers.

bpo-39932 is a good example: test_multiprocessing_fork fails with "test_multiprocessing_fork leaked [0, 2, 0] file descriptors". The test fails while run in paralle, but it also fails when re-run sequentially. Except that when I connect to the buildbot worker, it does not fail anymore.

test_multiprocessing_fork contains 356 test methods, the test file (Lib/test/ has 5741 lines of Python code, and the multiprocessing is made of 8149 lines of Python code and 1133 lines of C code. It's hard to audit such code. The multiprocessing uses multiple proceses, pipes, signals, etc. It's really hard to debug.

I propose to add an --bisect-failed option to test.regrtest to run test.bisect_cmd on failed tests. We can start to experiment it on Refleaks buildbots. Regular tests (not Refleaks tests) are easier to reproduce in general.

It should speedup analysis of reference leak and "altered environment" test failures. Having less test methods to audit is way simpler.

The implement should be that at the end of regrtest, after tests are re-run, run each failed test in test.bisect_cmd with the same command line arguments than test.regrtest.

test.bisect_cmd uses 100 iterations by default. It's ok if the bisection fails to reduce the number of test methods. At least, it should reduce the list in some cases.
msg365392 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-31 15:25
New changeset 01e743d704aeefd1c1825a6a96d0c766c5516fb1 by Victor Stinner in branch 'master':
bpo-40003: test.bisect_cmd copies Python options (GH-19246)
msg369256 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 15:51
While it would be nice, I'm not sure if it's a good idea to make buildbot jobs even longer. I wrote this issue to investigate an issue that I fixed in the meanwhile, so I close the issue.
Date User Action Args
2020-05-18 15:51:14vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg369256

stage: patch review -> resolved
2020-03-31 15:25:59vstinnersetmessages: + msg365392
2020-03-31 14:05:49vstinnersetkeywords: + patch
stage: patch review
pull_requests: + pull_request18604
2020-03-29 09:05:42shihai1991setnosy: + shihai1991
2020-03-28 05:56:09corona10setnosy: + corona10
2020-03-18 15:48:15vstinnercreate