Issue38296
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-09-27 17:26 by Kit Choi, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (7) | |||
---|---|---|---|
msg353382 - (view) | Author: Kit Yan Choi (Kit Choi) | Date: 2019-09-27 17:26 | |
I expect the following test to fail, because an "error" is not a "failure". Unexpectedly, the test passes: ``` class TestFailure(unittest.TestCase): @unittest.expectedFailure def test_expected_failure(self): raise TypeError() # for example, a typo. ``` ``` $ python -m unittest test_main x ---------------------------------------------------------------------- Ran 1 test in 0.000s OK (expected failures=1) ``` This behaviour exists since Python 2.7, and is still true for the Python 3.8.0b1 |
|||
msg353411 - (view) | Author: Terry J. Reedy (terry.reedy) * ![]() |
Date: 2019-09-27 21:00 | |
A function can fail to return an expected object by either returning a wrong object or raising an (unexpected) exception. The assertXyz methods, which ultimately raise AssertionError or something similar, are mostly about catching the first kind of failure, but tests should also catch and report the second kind. The traceback shows the kind of failure. The assertXyx failures add additional details after the traceback. import unittest class T(unittest.TestCase): def test_f(self): raise TypeError() unittest.main() # Properly results in Traceback (most recent call last): File "F:\Python\a\tem4.py", line 4, in test_f def test_f(self): raise TypeError() TypeError ---------------------------------------------------------------------- Ran 1 test in 0.050s FAILED (errors=1) |
|||
msg353413 - (view) | Author: Kit Yan Choi (Kit Yan Choi) * | Date: 2019-09-27 21:15 | |
For your test: class T(unittest.TestCase): def test_f(self): raise TypeError() If you run this test with unittest test runner, you should get this result: E ====================================================================== ERROR: test_f (test_main.T) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_main.py", line 5, in test_f def test_f(self): raise TypeError() TypeError ---------------------------------------------------------------------- Ran 1 test in 0.000s FAILED (errors=1) I expect to get this behaviour even if the test is decorated with unittest.expectedFailure. However, currently we get a success. Scenario: You create a class named Duck with a method "quack". Then you added a test, and test needs to call Duck.quack. Later on for whatever reason, you need to decorate the test with expectedFailure. The test passes with the expected failure. Then you rename the "quack" method to "walk", but you forget to update the test. Now the test is actually failing with an AttributeError, but you won't notice it because expectedFailure silences it. In this scenario, it is important to differentiate a "test error" and a "test failure". A test has four status: success, failure, error, skipped. I expect unittest.expectedFailure to make "failure" a "success" and a "success" a "failure", and it should leave "error" and "skipped" unchanged. Please consider reopening this issue. |
|||
msg353466 - (view) | Author: Kit Yan Choi (Kit Yan Choi) * | Date: 2019-09-28 17:14 | |
Pining Chris based on previous discussion in issue16997 ... Hope that's okay. I notice that the language in my initial message also conflates error and failure. My apologies on the carelessness. Just to clarify: @unittest.expectedFailure def test(self): THIS_VARIABLE_IS_UNDEFINED # ---> NameError should give: ERROR (errors=1) currently gives: OK (expected failures=1) By fixing this, we can help projects to maintain their tests decorated with expectedFailure so that the tests remaining to be meaningful. |
|||
msg353470 - (view) | Author: Terry J. Reedy (terry.reedy) * ![]() |
Date: 2019-09-28 18:29 | |
A test either passes or fails. Like a not operator, the expectedFailure decorator inverts the result. https://docs.python.org/3/library/unittest.html#unittest.expectedFailure @unittest.expectedFailure Mark the test as an expected failure. If the test fails it will be considered a success. If the test passes, it will be considered a failure. By itself, your 'test' method fails. Decorated, it should and does pass. As you suggested, using expectedFailure is a blunt instrument that can be misleading if not used carefully and not reviewed when editing the module tested. It is only used 7 times in test_xyz.py modules in the lib/test directory and subdirectories. |
|||
msg353474 - (view) | Author: Kit Yan Choi (Kit Yan Choi) * | Date: 2019-09-28 20:09 | |
I think Python does differentiate "test error" and "test failure" such that a test outcome state can be one of these: success, failure, error, skipped. One could refine these to six: expected success, unexpected success, expected failure, unexpected failure, error, skipped. For example, in the documentation for failureException: * failureException: determines which exception will be raised when the instance's assertion methods fail; test methods raising this exception will be deemed to have 'failed' rather than 'errored'. Another evidence: unittest.runner.TextTestResult, there are methods called "addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and "addUnexpectedSuccess". For example, this test outcome is marked as "FAILED": def test(self): x = 1 y = 2 self.assertEqual(x + y, 4) ====================================================================== FAIL: test (test_main.T) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_main.py", line 9, in test self.assertEqual(x + y, 4) AssertionError: 3 != 4 But the test outcome for this test is "ERROR": def test(self): x = 1 y = 2 + z # NameError self.assertEqual(x + y, 4) ====================================================================== ERROR: test (test_main.T) ---------------------------------------------------------------------- Traceback (most recent call last): File "test_main.py", line 8, in test y = 2 + z # NameError NameError: global name 'z' is not defined The issue here being "expectedFailure" converting "error" to "success", which is not expected, and is causing decorated tests to become unmaintained. While the rest of unittest differentiates "error" and "failure", expectedFailure does not. This is either a bug in the behaviour of expectedFailure, or a bug in the documentation for not being clear on the fact that unexpected error will be considered as expected failure (which I think is wrong). |
|||
msg353558 - (view) | Author: Kit Choi (Kit Choi2) | Date: 2019-09-30 09:20 | |
See issue38320 for documentation change request |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:20 | admin | set | github: 82477 |
2019-09-30 09:20:33 | Kit Choi2 | set | nosy:
+ Kit Choi2 messages: + msg353558 |
2019-09-28 20:09:49 | Kit Yan Choi | set | messages: + msg353474 |
2019-09-28 18:29:40 | terry.reedy | set | nosy:
+ ezio.melotti, michael.foord messages: + msg353470 versions: - Python 3.5, Python 3.6 |
2019-09-28 17:14:58 | Kit Yan Choi | set | nosy:
+ chris.jerdonek messages: + msg353466 |
2019-09-27 21:19:59 | Kit Yan Choi | set | nosy:
- Kit Choi |
2019-09-27 21:15:38 | Kit Yan Choi | set | nosy:
+ Kit Yan Choi messages: + msg353413 |
2019-09-27 21:00:16 | terry.reedy | set | status: open -> closed nosy: + terry.reedy messages: + msg353411 resolution: not a bug stage: resolved |
2019-09-27 17:26:09 | Kit Choi | create |