Message353474
I think Python does differentiate "test error" and "test failure" such that a test outcome state can be one of these: success, failure, error, skipped. One could refine these to six: expected success, unexpected success, expected failure, unexpected failure, error, skipped.
For example, in the documentation for failureException:
* failureException: determines which exception will be raised when
the instance's assertion methods fail; test methods raising this
exception will be deemed to have 'failed' rather than 'errored'.
Another evidence: unittest.runner.TextTestResult, there are methods called "addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and "addUnexpectedSuccess".
For example, this test outcome is marked as "FAILED":
def test(self):
x = 1
y = 2
self.assertEqual(x + y, 4)
======================================================================
FAIL: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_main.py", line 9, in test
self.assertEqual(x + y, 4)
AssertionError: 3 != 4
But the test outcome for this test is "ERROR":
def test(self):
x = 1
y = 2 + z # NameError
self.assertEqual(x + y, 4)
======================================================================
ERROR: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test_main.py", line 8, in test
y = 2 + z # NameError
NameError: global name 'z' is not defined
The issue here being "expectedFailure" converting "error" to "success", which is not expected, and is causing decorated tests to become unmaintained. While the rest of unittest differentiates "error" and "failure", expectedFailure does not. This is either a bug in the behaviour of expectedFailure, or a bug in the documentation for not being clear on the fact that unexpected error will be considered as expected failure (which I think is wrong). |
|
Date |
User |
Action |
Args |
2019-09-28 20:09:49 | Kit Yan Choi | set | recipients:
+ Kit Yan Choi, terry.reedy, ezio.melotti, michael.foord, chris.jerdonek |
2019-09-28 20:09:49 | Kit Yan Choi | set | messageid: <1569701389.42.0.683405826262.issue38296@roundup.psfhosted.org> |
2019-09-28 20:09:49 | Kit Yan Choi | link | issue38296 messages |
2019-09-28 20:09:49 | Kit Yan Choi | create | |
|