This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author zach.ware
Recipients ezio.melotti, michael.foord, pitrou, rbcollins, zach.ware
Date 2015-12-17.05:49:34
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1450331375.94.0.689081172202.issue25894@psf.upfronthosting.co.za>
In-reply-to
Content
The title can barely be called accurate; the description of the problem isn't easy to condense to title length.  Here's the issue:

$ cat subtest_test.py 
import os
import unittest

class TestClass(unittest.TestCase):

    def test_subTest(self):
        for t in map(int, os.environ.get('tests', '1')):
            with self.subTest(t):
                if t > 1:
                    raise unittest.SkipTest('skipped')
                self.assertTrue(t)

if __name__ == '__main__':
    unittest.main()
$ ./python.exe subtest_test.py 
.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK
$ tests=01 ./python.exe subtest_test.py 

======================================================================
FAIL: test_subTest (__main__.TestClass) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtest_test.py", line 12, in test_subTest
    self.assertTrue(t)
AssertionError: 0 is not true

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
$ tests=012 ./python.exe subtest_test.py 
s
======================================================================
FAIL: test_subTest (__main__.TestClass) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtest_test.py", line 12, in test_subTest
    self.assertTrue(t)
AssertionError: 0 is not true

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1, skipped=1)



Note that on the first run, the short summary is ".", as expected.  The second is "", when one of the subTests fails, but then the third is "s", when one subtest fails but another is skipped.  This also extends to verbose mode:

$ ./python.exe subtest_test.py -v
test_subTest (__main__.TestClass) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.001s

OK
$ tests=01 ./python.exe subtest_test.py -v
test_subTest (__main__.TestClass) ... 
======================================================================
FAIL: test_subTest (__main__.TestClass) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtest_test.py", line 12, in test_subTest
    self.assertTrue(t)
AssertionError: 0 is not true

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1)
$ tests=012 ./python.exe subtest_test.py -v
test_subTest (__main__.TestClass) ... skipped 'skipped'

======================================================================
FAIL: test_subTest (__main__.TestClass) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "subtest_test.py", line 12, in test_subTest
    self.assertTrue(t)
AssertionError: 0 is not true

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (failures=1, skipped=1)


Note the first run shows "... ok", the second "... ", and the third "... skipped 'skipped'"


I'm unsure what the solution should be.  There should at least be some indication that the test finished, but should mixed results be reported as 'm' ("mixed results" in verbose mode), or should failure/error take precedence, or should every different result be represented?
History
Date User Action Args
2015-12-17 05:49:36zach.waresetrecipients: + zach.ware, pitrou, rbcollins, ezio.melotti, michael.foord
2015-12-17 05:49:35zach.waresetmessageid: <1450331375.94.0.689081172202.issue25894@psf.upfronthosting.co.za>
2015-12-17 05:49:35zach.warelinkissue25894 messages
2015-12-17 05:49:34zach.warecreate