classification
Title: Regression tests with -w should only re-run affected test methods, not the entire file
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: lukasz.langa Nosy List: lukasz.langa, miss-islington, pablogsal
Priority: normal Keywords: patch

Created on 2021-07-22 14:32 by lukasz.langa, last changed 2021-07-22 21:18 by lukasz.langa. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 27287 merged lukasz.langa, 2021-07-22 14:36
PR 27290 merged miss-islington, 2021-07-22 19:00
PR 27293 merged lukasz.langa, 2021-07-22 20:42
Messages (4)
msg397985 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-22 14:32
When the CPython test suite re-runs a flaky failed test, it doesn’t actually re-run just that failed test but the entire test file. The most common case for flaky tests is when networking, threading, or multiprocessing is involved. Frustratingly those are some of the largest test files we have. Re-running them takes a lot of time.

Instead of re-running the entire file, regrtest should only re-run what actually failed.

NOTE: I added 3.10 and 3.9 to this issue even though it's not a pure bugfix. The reason is that this can visibly speed up CI for open pull requests.

(Of course, it would be best to avoid flaky tests altogether and we’re working on that, but re-running what failed is a pragmatic stopgap that is necessary in times of distributed CI that we don’t fully control.)
msg397994 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-22 18:26
New changeset f1afef5e0d93d66fbf3c9aaeab8b3b8da9617583 by Łukasz Langa in branch 'main':
bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287)
https://github.com/python/cpython/commit/f1afef5e0d93d66fbf3c9aaeab8b3b8da9617583
msg398001 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-22 20:31
New changeset 9608719e12b14087c1cb34f77c847974442cd6e7 by Miss Islington (bot) in branch '3.10':
bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287) (GH-27290)
https://github.com/python/cpython/commit/9608719e12b14087c1cb34f77c847974442cd6e7
msg398007 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2021-07-22 21:17
New changeset 0f55d21212a0fb6dec4479eb76249ab2b54e57a3 by Łukasz Langa in branch '3.9':
[3.9] bpo-44708: Only re-run test methods that match names of previously failing test methods (GH-27287) (GH-27293)
https://github.com/python/cpython/commit/0f55d21212a0fb6dec4479eb76249ab2b54e57a3
History
Date User Action Args
2021-07-22 21:18:18lukasz.langasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-07-22 21:17:23lukasz.langasetmessages: + msg398007
2021-07-22 20:42:21lukasz.langasetpull_requests: + pull_request25836
2021-07-22 20:31:06lukasz.langasetmessages: + msg398001
2021-07-22 19:00:06miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request25832
2021-07-22 18:26:01lukasz.langasetmessages: + msg397994
2021-07-22 14:36:54lukasz.langasetkeywords: + patch
pull_requests: + pull_request25829
2021-07-22 14:32:29lukasz.langacreate