classification
Title: unittest expectedFailure does not differentiate errors from failures
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: Kit Choi2, Kit Yan Choi, chris.jerdonek, ezio.melotti, michael.foord, terry.reedy
Priority: normal Keywords:

Created on 2019-09-27 17:26 by Kit Choi, last changed 2019-09-30 09:20 by Kit Choi2. This issue is now closed.

Messages (7)
msg353382 - (view) Author: Kit Yan Choi (Kit Choi) Date: 2019-09-27 17:26
I expect the following test to fail, because an "error" is not a "failure".
Unexpectedly, the test passes:

```
class TestFailure(unittest.TestCase):

    @unittest.expectedFailure
    def test_expected_failure(self):
        raise TypeError()   # for example, a typo.
```

```
$ python -m unittest test_main
x
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK (expected failures=1)
```

This behaviour exists since Python 2.7, and is still true for the Python 3.8.0b1
msg353411 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-09-27 21:00
A function can fail to return an expected object by either returning a wrong object or raising an (unexpected) exception.  The assertXyz methods, which ultimately raise AssertionError or something similar, are mostly about catching the first kind of failure, but tests should also catch and report the second kind.  The traceback shows the kind of failure.  The assertXyx failures add additional details after the traceback.


import unittest

class T(unittest.TestCase):
    def test_f(self): raise TypeError()

unittest.main()

# Properly results in

Traceback (most recent call last):
  File "F:\Python\a\tem4.py", line 4, in test_f
    def test_f(self): raise TypeError()
TypeError

----------------------------------------------------------------------
Ran 1 test in 0.050s

FAILED (errors=1)
msg353413 - (view) Author: Kit Yan Choi (Kit Yan Choi) * Date: 2019-09-27 21:15
For your test:

class T(unittest.TestCase):
    def test_f(self): raise TypeError()

If you run this test with unittest test runner, you should get this result:

E
======================================================================
ERROR: test_f (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_main.py", line 5, in test_f
    def test_f(self): raise TypeError()
TypeError

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)


I expect to get this behaviour even if the test is decorated with unittest.expectedFailure. However, currently we get a success.


Scenario:


You create a class named Duck with a method "quack".  Then you added a test, and test needs to call Duck.quack.

Later on for whatever reason, you need to decorate the test with expectedFailure.  The test passes with the expected failure.

Then you rename the "quack" method to "walk", but you forget to update the test.  Now the test is actually failing with an AttributeError, but you won't notice it because expectedFailure silences it.

In this scenario, it is important to differentiate a "test error" and a "test failure". A test has four status: success, failure, error, skipped.  I expect unittest.expectedFailure to make "failure" a "success" and a "success" a "failure", and it should leave "error" and "skipped" unchanged.

Please consider reopening this issue.
msg353466 - (view) Author: Kit Yan Choi (Kit Yan Choi) * Date: 2019-09-28 17:14
Pining Chris based on previous discussion in issue16997 ... Hope that's okay.

I notice that the language in my initial message also conflates error and failure. My apologies on the carelessness.

Just to clarify:


    @unittest.expectedFailure
    def test(self):
        THIS_VARIABLE_IS_UNDEFINED  # ---> NameError


should give: ERROR (errors=1)
currently gives: OK (expected failures=1)


By fixing this, we can help projects to maintain their tests decorated with expectedFailure so that the tests remaining to be meaningful.
msg353470 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-09-28 18:29
A test either passes or fails.  Like a not operator, the expectedFailure decorator inverts the result.

https://docs.python.org/3/library/unittest.html#unittest.expectedFailure
@unittest.expectedFailure
    Mark the test as an expected failure. If the test fails it will be considered a success. If the test passes, it will be considered a failure.

By itself, your 'test' method fails.  Decorated, it should and does pass.

As you suggested, using expectedFailure is a blunt instrument that can be misleading if not used carefully and not reviewed when editing the module tested.  It is only used 7 times in test_xyz.py modules in the lib/test directory and subdirectories.
msg353474 - (view) Author: Kit Yan Choi (Kit Yan Choi) * Date: 2019-09-28 20:09
I think Python does differentiate "test error" and "test failure" such that a test outcome state can be one of these: success, failure, error, skipped. One could refine these to six: expected success, unexpected success, expected failure, unexpected failure, error, skipped.


For example, in the documentation for failureException:

    * failureException: determines which exception will be raised when
        the instance's assertion methods fail; test methods raising this
        exception will be deemed to have 'failed' rather than 'errored'.


Another evidence: unittest.runner.TextTestResult, there are methods called "addSuccess", "addError", "addFailure", "addSkip", "addExpectedFailure" and "addUnexpectedSuccess".


For example, this test outcome is marked as "FAILED":

def test(self):
    x = 1
    y = 2
    self.assertEqual(x + y, 4)


======================================================================
FAIL: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_main.py", line 9, in test
    self.assertEqual(x + y, 4)
AssertionError: 3 != 4


But the test outcome for this test is "ERROR":

    def test(self):
        x = 1
        y = 2 + z  # NameError                                                  
        self.assertEqual(x + y, 4)


======================================================================
ERROR: test (test_main.T)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_main.py", line 8, in test
    y = 2 + z  # NameError
NameError: global name 'z' is not defined


The issue here being "expectedFailure" converting "error" to "success", which is not expected, and is causing decorated tests to become unmaintained. While the rest of unittest differentiates "error" and "failure", expectedFailure does not. This is either a bug in the behaviour of expectedFailure, or a bug in the documentation for not being clear on the fact that unexpected error will be considered as expected failure (which I think is wrong).
msg353558 - (view) Author: Kit Choi (Kit Choi2) Date: 2019-09-30 09:20
See issue38320 for documentation change request
History
Date User Action Args
2019-09-30 09:20:33Kit Choi2setnosy: + Kit Choi2
messages: + msg353558
2019-09-28 20:09:49Kit Yan Choisetmessages: + msg353474
2019-09-28 18:29:40terry.reedysetnosy: + ezio.melotti, michael.foord

messages: + msg353470
versions: - Python 3.5, Python 3.6
2019-09-28 17:14:58Kit Yan Choisetnosy: + chris.jerdonek
messages: + msg353466
2019-09-27 21:19:59Kit Yan Choisetnosy: - Kit Choi
2019-09-27 21:15:38Kit Yan Choisetnosy: + Kit Yan Choi
messages: + msg353413
2019-09-27 21:00:16terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg353411

resolution: not a bug
stage: resolved
2019-09-27 17:26:09Kit Choicreate