This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: unexpected successes are not output
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.11
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: ezio.melotti, iritkatriel, michael.foord, rbcollins, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-11-07 10:21 by rbcollins, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 30138 merged serhiy.storchaka, 2021-12-16 10:56
Messages (11)
msg230783 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-11-07 10:21
Unexpected successes cause failures, but we don't output details about them at the end of the run.

From https://code.google.com/p/unittest-ext/issues/detail?id=22

A complicating factor is that we don't have a backtrace to show - but we may have captured stdout/stderr or in future various test attachments to show.
msg407833 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-06 16:27
I think this has been fixed:

% cat mm.py 
import unittest

class TestStringMethods(unittest.TestCase):

    @unittest.expectedFailure
    def test_upper(self):
        self.assertEqual(2, 2)

if __name__ == '__main__':
    unittest.main()




% ./python.exe mm.py 
x
----------------------------------------------------------------------
Ran 1 test in 0.001s

OK (expected failures=1)
iritkatriel@Irits-MBP cpython % vi mm.py          
iritkatriel@Irits-MBP cpython % ./python.exe mm.py
u
----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (unexpected successes=1)
msg407834 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-06 16:28
Or do you mean that there should be more output?
msg408679 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-16 09:46
In verbose mode we do get output like the one in the code.google issue:

% cat tmp.py

import unittest

class TestStringMethods(unittest.TestCase):

    @unittest.expectedFailure
    def test_upper(self):
        self.assertEqual(2, 2)

    def test_lower(self):
        return

if __name__ == '__main__':
    unittest.main()

% ./python.exe -m tmp -v
test_lower (__main__.TestStringMethods) ... ok
test_upper (__main__.TestStringMethods) ... unexpected success
msg408681 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-16 09:48
I don't think adding more information from the test would be helpful - when a test fails you have (and need) information about what happened. But when a test succeeds the code of the test basically tells you what happened - all the assertions passed.

I think we can close this.
msg408683 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-12-16 10:15
I think the OP means that test details (test description, traceback, captured output) are printed for ERROR and FAIL in TextTestResult.printErrors(). Other possible results besides error and failure are success, skipped, expected failure and unexpected success. Success, skipped and expected failure are expected, they do not cause total failure, but unexpected success is different. Result.wasSuccessful() returns False if there are any errors, failures or unexpected successes. But unlike errors and failures we do not have any details about unexpected successes. It looks like oversight to me.

Yes, there are no traceback for unexpected success, but at least we can output the test description and captured output.
msg408685 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-12-16 10:57
There are no other details for unexpected successes, only test descriptions.
msg408689 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-16 11:37
Yes, this is why I don’t understand the request here. When a test fails you know which assertion was violated or which line raised an exception. When a multi line test succeeds, you don’t know which line was supposed to fail.
msg408690 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-12-16 11:42
I know that in unittest2 an unexpected success did not cause the test to fail (when we moved from unittest2 to unittest at work we got test failures due to unexpected successes that previously did not show up). I don't know the whole history of how unittest2 came to be, I think it was a clone of an older copy of unittest? Maybe that it the background for this issue?
msg408693 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-12-16 11:52
Unless you tun tests in verbose mode you do not know which of tests was unexpectedly successful. And even in the verbose mode it is not easy to find that test in long output.

Yes, unexpected successes considered successes in earlier versions. See issue20165.

You can find many interesting when read git logs.
msg409190 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-12-26 11:22
New changeset 1944434b44e0118e812bf63f47b268ff6dd0c8f1 by Serhiy Storchaka in branch 'main':
bpo-22815: Print unexpected successes in summary in TextTestResult (GH-30138)
https://github.com/python/cpython/commit/1944434b44e0118e812bf63f47b268ff6dd0c8f1
History
Date User Action Args
2022-04-11 14:58:09adminsetgithub: 67004
2021-12-26 11:23:43serhiy.storchakasetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2021-12-26 11:22:47serhiy.storchakasetmessages: + msg409190
2021-12-16 11:52:50serhiy.storchakasetmessages: + msg408693
2021-12-16 11:42:13iritkatrielsetmessages: + msg408690
2021-12-16 11:37:25iritkatrielsetmessages: + msg408689
2021-12-16 10:57:51serhiy.storchakasettype: behavior -> enhancement
messages: + msg408685
2021-12-16 10:56:20serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request28356
2021-12-16 10:15:44serhiy.storchakasetstatus: pending -> open

assignee: serhiy.storchaka
versions: + Python 3.11, - Python 3.5
nosy: + ezio.melotti, serhiy.storchaka, michael.foord

messages: + msg408683
resolution: out of date -> (no value)
2021-12-16 09:48:42iritkatrielsetstatus: open -> pending

messages: + msg408681
2021-12-16 09:46:55iritkatrielsetmessages: + msg408679
2021-12-06 16:28:45iritkatrielsetstatus: pending -> open

messages: + msg407834
2021-12-06 16:27:50iritkatrielsetstatus: open -> pending

nosy: + iritkatriel
messages: + msg407833

resolution: out of date
2014-11-07 10:21:48rbcollinscreate