classification
Title: "unittest.TestCase.debug" should honour "skip" (and other test controls)
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: 41620 Superseder:
Assigned To: Nosy List: dmaurer, ezio.melotti, lisroach, michael.foord, miss-islington, rbcollins, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2019-04-20 09:20 by dmaurer, last changed 2021-09-18 14:06 by dmaurer.

Files
File name Uploaded Description Edit
utest.py dmaurer, 2019-04-27 08:56
Pull Requests
URL Status Linked Edit
PR 13180 open lisroach, 2019-05-07 21:32
PR 28446 merged serhiy.storchaka, 2021-09-18 12:11
PR 28447 merged miss-islington, 2021-09-18 12:34
PR 28448 merged miss-islington, 2021-09-18 12:34
Messages (15)
msg340569 - (view) Author: Dieter Maurer (dmaurer) Date: 2019-04-20 09:20
Currently, "TestCase.run" supports several features to control testing - among others, a test can be skipped via the attribute "__unittest_skip__". "TestCase.debug" ignores all those controls and calls the test method unconditionally.

I am using "zope.testrunner" to run test suites. Its "-D" option switches from "TestCase.run" to "TestCase.debug" in order to allow the analysis of the state of a failing test in the Python debugger. "-D" is typically used if a test in a larger suite failed and a detailed analysis is required to determine the failure's cause. It is important that this second run executes the same tests as the first run; it is not helpful when the second run fails in a test skipped in the first run. Therefore, "TestCase.debug" should honour all test controls supported by  "TestCase.run".

One could argue that the testsuite runner should implement this logic. However, this would force the runner to duplicate the test control logic using internal implementation details of "unittest". Conceptually, it is much nicer to have the test control encapsulated by "unittest".
msg340954 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-27 00:48
3.6 only gets security fixes.  This appears to be an enhancement request.  SkipTest is an exception and according to the doc it should be propagated to the caller.  __unittest_skip__ is an undocumented internal name, so you probably should not be using it.

If this request is not rejected, we will need test code demonstrating the issue *that does not use zope*.
msg340976 - (view) Author: Dieter Maurer (dmaurer) Date: 2019-04-27 08:56
Terry J. Reedy wrote:
> SkipTest is an exception and according to the doc it should be propagated to the caller.  __unittest_skip__ is an undocumented internal name, so you probably should not be using it.
>
> If this request is not rejected, we will need test code demonstrating the issue *that does not use zope*.

The "skip" I am speaking about is a decorator for tests or test classes telling a "runner" to skip tests (likely because conditions for their success are not met - usually, that required packages are not installed). This "skip" (decorator) uses `__unittest_skip__` to tell the  the "unittest runner" (that is `unittest.TestCase.run`) that the test should be skipped.

The "unittest runner" is extremely rudimentary and lacks many important features. That is why I use the much more feature rich `zope.testrunner`. I could implement the `__unittest_skip__` logic there - but as you wrote - this should remain encapsulated inside `unittest` (that's why I wrote this issue).

I attach a test script - which likely is not convincing as the feature is mainly important for a sophisticated test runner, not for explicit calls of `TestCase.debug`.
msg341122 - (view) Author: Lisa Roach (lisroach) * (Python committer) Date: 2019-04-29 21:29
+1, I think tests should run the same way in debug() and run(), the difference being limited only to how the result is handled. 

Marking a test as skipped should still be skipped while debugging, else it pollutes the results with tests that will not run with run().
msg341135 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2019-04-30 04:15
1. NO SKIP utest.py with debug prints added and @skipIf initially commented out.
---
from unittest import TestCase, skipIf

#@skipIf(True, "Skip Testing")
class Tests(TestCase):
  def test_skip(self):
    "this test will fail - if not skipped"
    print('asserting')
    self.assertEqual(0, 1)

print(Tests("test_skip").run())
print(Tests("test_skip").debug())
---

test_skip is run twice, with the output difference being as documented.

asserting
<unittest.result.TestResult run=1 errors=0 failures=1>
asserting
Traceback (most recent call last):
...
AssertionError: 0 != 1


2. SKIPTEST Adding "self.skipTest('why')" either in a new setUp() or at the top of skip_test() and adding another print 

print(Tests("test_skip").run().skipped)

results in the output I expect from the doc.

<unittest.result.TestResult run=1 errors=0 failures=0>
[(<__main__.Tests testMethod=test_skip>, 'why')]
Traceback (most recent call last):
...
unittest.case.SkipTest: why


3. SKIPIF CLASS Uncommenting @skipIf (the OP's case) instead results in

None
asserting
Traceback ...
AssertionError: 0 != 1

Since .run does not run the test, I agree that debug() running the test and raising AssertionError is a bug.  The test should not be run.  In my original comments, I expected SkipTest, as in cases 2 above and 4 below.  I have not yet checked the current tests.


4. SKIPIF FUNC Moving the skipIf decorator to test_skip results in
None
Traceback (most recent call last):
..
unittest.case.SkipTest: Skip Testing

I don't know why run() returns None for skipIf cases instead of returning a TestResult with non-empty skipped, as it does for skipTest, or if the None is a separate bug.
msg348576 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2019-07-28 13:26
"I don't know why run() returns None for skipIf cases instead of returning a TestResult with non-empty skipped, as it does for skipTest, or if the None is a separate bug."

That does sound like a bug.
msg400560 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-08-29 21:11
See issue41620 for run() returning None.
msg401105 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-05 19:07
There are too many issues with TestResult.debug(). It does not call tearDown() and doCleanups() if the test is skipped, failed or raises other exception (including errors in cleanup). It does not support skipping decorators. It does not support the expectedFailure decorator. It does not work at all in IsolatedAsyncioTestCase.
msg402127 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-18 12:26
PR 28446 makes TestCase.debug() honoring the skipping decorator.

The difference with PR 13180:

1. It does not run setUp() and tearDown() for tests decorated with the skipping decorator, as TestCase.run().
2. It has the same behavior for tests decorated with the skipping decorator and tests using skipTest() or raising a SkipTest -- raises a SkipTest.
3. Tests actually test the skipping decorator.
msg402128 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-18 12:31
Since the behavior of debug() was different for the following cases:

    @skip('skipping')
    def test1(self):
        pass
    @othedecorator
    @skip('skipping')
    def test2(self):
        pass
    def test3(self):
        self.skipTest('skipping')

I consider it as a bug and will backport the fix.
msg402129 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-18 12:34
New changeset dea59cf88adf5d20812edda330e085a4695baba4 by Serhiy Storchaka in branch 'main':
bpo-36674: Honour the skipping decorators in TestCase.debug() (GH-28446)
https://github.com/python/cpython/commit/dea59cf88adf5d20812edda330e085a4695baba4
msg402130 - (view) Author: miss-islington (miss-islington) Date: 2021-09-18 13:22
New changeset 753f7af22e78456726496fd5d8bf38e7e6567e4e by Miss Islington (bot) in branch '3.10':
bpo-36674: Honour the skipping decorators in TestCase.debug() (GH-28446)
https://github.com/python/cpython/commit/753f7af22e78456726496fd5d8bf38e7e6567e4e
msg402131 - (view) Author: miss-islington (miss-islington) Date: 2021-09-18 13:22
New changeset 7e465a6b8273dc0b6cb484c65664f618afa22484 by Miss Islington (bot) in branch '3.9':
bpo-36674: Honour the skipping decorators in TestCase.debug() (GH-28446)
https://github.com/python/cpython/commit/7e465a6b8273dc0b6cb484c65664f618afa22484
msg402133 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-09-18 14:02
See issue45238 for debug() in IsolatedAsyncioTestCase.
msg402134 - (view) Author: Dieter Maurer (dmaurer) Date: 2021-09-18 14:06
Thank you for working on this issue!
History
Date User Action Args
2021-09-18 14:06:05dmaurersetmessages: + msg402134
2021-09-18 14:02:21serhiy.storchakasetmessages: + msg402133
2021-09-18 13:22:43miss-islingtonsetmessages: + msg402131
2021-09-18 13:22:35miss-islingtonsetmessages: + msg402130
2021-09-18 12:34:35miss-islingtonsetpull_requests: + pull_request26851
2021-09-18 12:34:31miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request26850
2021-09-18 12:34:26serhiy.storchakasetmessages: + msg402129
2021-09-18 12:31:22serhiy.storchakasetmessages: + msg402128
2021-09-18 12:26:05serhiy.storchakasettype: enhancement -> behavior
messages: + msg402127
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 3.8
2021-09-18 12:11:37serhiy.storchakasetpull_requests: + pull_request26849
2021-09-05 19:07:41serhiy.storchakasetmessages: + msg401105
2021-08-29 21:11:23serhiy.storchakasetnosy: + serhiy.storchaka
dependencies: + Python Unittest does not return results object when the test is skipped
messages: + msg400560
2021-08-29 21:02:02serhiy.storchakasetpull_requests: - pull_request14706
2019-07-28 13:26:47michael.foordsetmessages: + msg348576
2019-07-24 17:42:52pradyunsgsetpull_requests: + pull_request14706
2019-05-07 21:33:10lisroachsetpull_requests: - pull_request13094
2019-05-07 21:32:40lisroachsetpull_requests: + pull_request13096
2019-05-07 21:05:56lisroachsetkeywords: + patch
stage: test needed -> patch review
pull_requests: + pull_request13094
2019-04-30 04:15:51terry.reedysetmessages: + msg341135
2019-04-29 21:29:50lisroachsetnosy: + lisroach
messages: + msg341122
2019-04-27 08:56:29dmaurersetfiles: + utest.py

messages: + msg340976
2019-04-27 00:48:53terry.reedysetversions: + Python 3.8, - Python 3.6
nosy: + terry.reedy

messages: + msg340954

type: behavior -> enhancement
stage: test needed
2019-04-20 09:47:50xtreaksetnosy: + rbcollins, ezio.melotti, michael.foord
2019-04-20 09:20:54dmaurercreate