classification
Title: Error during test case and tearDown
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ezio.melotti Nosy List: beng94, bennoleslie, ezio.melotti, flyjwayur, michael.foord, python-dev, rbcollins
Priority: normal Keywords: easy, patch

Created on 2015-11-21 07:54 by bennoleslie, last changed 2016-03-13 07:44 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
issue25687.diff flyjwayur, 2016-03-12 11:30 review
Messages (6)
msg255045 - (view) Author: Benno Leslie (bennoleslie) * Date: 2015-11-21 07:54
I'm not sure if this should be considered a bug, but the behaviour is surprising and not clearly documented.

I a have a very simple test that has an error during both the test case, and during tearDown.

"""
import unittest

class Test(unittest.TestCase):

    def test_a(self):
        asdf

    def tearDown(self):
        asdf

if __name__ == '__main__':
    unittest.main()
"""

When this occurs it is surprising (to me) that the output is:

"""
Ran 1 test in 0.000s

FAILED (errors=2)
"""

In particular, the fact that has more errors than there are tests that have been run. Obviously in this very simple example it is clear what has happened, however in a test suite that has hundreds of test cases it is somewhat frustrating to have the number of failing test cases over-reported. (And of course in the real-world test suite that led to this the tearDown doesn't fail on every single test case like in this simplified example).

Although there are definitely two errors occurring in my example, in all other cases, only the first error would be reported. e.g.: an error in setUp wouldn't run the test case, and only the first (potential) error in the testcase itself would occur.

I think that either:

1/ The documentation of the tearDown method should clearly indicate that an error in tearDown would cause multiple errors to be reported for the single testCase, or.
2/ Change the implementation so that if there is an exception during tearDown, and there was already an error in the test case only the first error is reported.

2 is probably a non-starter given backwards compatibility concerns (even assuming anyone else thinks this is the best behaviour in this corner case).

I'd be happy to try and draft something for #1 if that is the best action.
msg257243 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2015-12-31 00:26
This happens because TestCase.run (Lib/unittest/case.py:595) runs setUp/test/tearDown in 3 separate testPartExecutor context manager  (Lib/unittest/case.py:54).  testPartExecutor appends any error to self.errors (Lib/unittest/case.py:72) and TextTestRunner.run simply reports the total number of errors Lib/unittest/runner.py:204).
If an error happens in the setUp, the test and tearDown are not executed, but if it happens in the test, the tearDown is still executed, possibly appending a second error to self.errors.

I don't see any easy way to fix this, since both errors should stay in self.errors and be reported, so removing one is not an option.  Trying to determine if 2 errors are related to a single test/tearDown pair in the TestRunner might be possible, but probably not worth it.

Adding a sentence to the doc and possibly a comment in TestCase.run to document this corner case is probably enough.
msg258693 - (view) Author: Tamás Bence Gedai (beng94) * Date: 2016-01-20 16:57
I think it's in the docs, although if you don't find it sufficient I might try to update it.

`tearDown method <https://docs.python.org/3.5/library/unittest.html#unittest.TestCase.tearDown>`_
"This is called even if the test method raised an exception [...] Any exception, other than AssertionError or SkipTest, raised by this method will be considered an error rather than a test failure. This method will only be called if the setUp() succeeds, regardless of the outcome of the test method."
msg261645 - (view) Author: HyeSoo Park (flyjwayur) * Date: 2016-03-12 11:30
I added 'additional' and (thus increasing the total number of error counts.) to 'teardown' explanation of the document to make it more clear.
msg261683 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-13 07:42
New changeset cecd39887faa by Ezio Melotti in branch '3.5':
#25687: clarify that errors in tearDown increase the total number of reported errors.  Initial patch by HyeSoo Park.
https://hg.python.org/cpython/rev/cecd39887faa

New changeset d5f5a6f514f2 by Ezio Melotti in branch 'default':
#25687: merge with 3.5.
https://hg.python.org/cpython/rev/d5f5a6f514f2

New changeset 5e64ffe576a9 by Ezio Melotti in branch '2.7':
#25687: clarify that errors in tearDown increase the total number of reported errors.  Initial patch by HyeSoo Park.
https://hg.python.org/cpython/rev/5e64ffe576a9
msg261684 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2016-03-13 07:44
Fixed, thanks for the report and the patch!
History
Date User Action Args
2016-03-13 07:44:00ezio.melottisetstatus: open -> closed
resolution: fixed
messages: + msg261684

stage: patch review -> resolved
2016-03-13 07:42:56python-devsetnosy: + python-dev
messages: + msg261683
2016-03-12 15:09:18ezio.melottisetassignee: ezio.melotti
stage: needs patch -> patch review
2016-03-12 11:30:51flyjwayursetfiles: + issue25687.diff

nosy: + flyjwayur
messages: + msg261645

keywords: + patch
2016-01-20 16:57:43beng94setnosy: + beng94
messages: + msg258693
2015-12-31 00:26:33ezio.melottisetkeywords: + easy

stage: needs patch
messages: + msg257243
versions: - Python 3.4
2015-11-27 18:17:20terry.reedysetnosy: + rbcollins, ezio.melotti, michael.foord

versions: + Python 3.5, Python 3.6, - Python 3.2, Python 3.3
2015-11-21 07:54:27bennolesliecreate