This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Test suite skips failing tests when setUp[Class] fails
Type: behavior Stage: patch review
Components: Tests Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, gvanrossum, kj, sobolevn
Priority: normal Keywords: patch

Created on 2022-01-25 15:42 by sobolevn, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 30895 open sobolevn, 2022-01-25 20:48
Messages (2)
msg411620 - (view) Author: Nikita Sobolev (sobolevn) * (Python triager) Date: 2022-01-25 15:42
Here's what happened. We had an error in `test_typing.py`, which was silently ignored.

```
 ======================================================================
ERROR: setUpClass (test.test_typing.NewTypeTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "D:\a\cpython\cpython\lib\test\test_typing.py", line 3917, in setUpClass
    UserId = NewType('UserId', int)
NameError: name 'NewType' is not defined

----------------------------------------------------------------------
Ran 396 tests in 0.085s

FAILED (errors=1, skipped=1)
test test_typing failed
```

Link: https://github.com/python/cpython/runs/4902363883?check_suite_focus=true

But, later the suite runner tried to rerun it:

```
0:09:12 load avg: 6.37 Re-running failed tests in verbose mode
0:09:12 load avg: 6.37 Re-running test_typing in verbose mode (matching: setUpClass)

1 re-run test:
    test_typing

1 test run no tests:
    test_typing
```

And since nothing matched `setUpClass` - no tests were executed and the CI went green instead of red.

What can we do?
1. Only schedule real `test_` item to be rerun, fail for everything else
2. Convert `setupClass` failure into the whole class rerun
3. Other options?

I would like to work on this, when we will decide which way is best.
msg411625 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2022-01-25 16:03
I vote for option 2.
History
Date User Action Args
2022-04-11 14:59:55adminsetgithub: 90681
2022-01-25 20:48:08sobolevnsetkeywords: + patch
stage: patch review
pull_requests: + pull_request29076
2022-01-25 16:04:14kjsettitle: Test suite skips failing tests -> Test suite skips failing tests when setUp[Class] fails
2022-01-25 16:03:48kjsetnosy: + ezio.melotti, kj, gvanrossum
messages: + msg411625
2022-01-25 15:42:34sobolevncreate