This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: unittest increment tests executed
Type: enhancement Stage:
Components: Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: chelmite, iritkatriel, rhettinger
Priority: normal Keywords:

Created on 2021-04-16 05:39 by chelmite, last changed 2022-04-11 14:59 by admin.

Messages (4)
msg391166 - (view) Author: Steve Kelem (chelmite) Date: 2021-04-16 05:39
The unittest module (and most other unittest modules) keep a count of the number of functional tests run, as defined by the number of times testXXXX is called.

I would like the option to count the number of times that assertXXXX is called.

I have to run many tests (in unittest's sense of the word), but each of the tests consists of iterating through an array of hundreds or thousands of individual test cases with their expected results. These check for corner cases, interior, exterior cases, and for mathematical accuracy. It doesn't make sense to make a separate test case for each test value, when iterating through an array of values, and calling AssertEquals (or whatever is called for) for each expected & actual value-pair.
Yes, the existing methodology tells me that a particular functional test works, but it is important to know whether it passed based on a single test value (because the test development is not done or the tester was lazy) or whether it passed thousands of tests and will give adequate coverage. (Yes that can be cheated too, but assume that the tests are developed in earnest.)
msg391168 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-04-16 06:26
ISTM the call count is too indirect to provide a useful indicator of coverage.
msg391273 - (view) Author: Steve Kelem (chelmite) Date: 2021-04-17 07:34
Technically, you're right, however each domain his its own notion of what "adequate coverage" means. Some projects use software code coverage. I deal with integrated circuit observability/controllability, and stocastic functional testing.

Until something, outside of the scope of unittest, becomes available (or affordable) having a count of the number of test points for each test gives me a good idea of what testing has been done and what needs to be expanded. Practically, having a count of the number of tests is a first-order useful measure for many of the projects I deal with.
msg391317 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2021-04-17 22:51
See the section on "Distinguishing test iterations using subtests"

https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests
History
Date User Action Args
2022-04-11 14:59:44adminsetgithub: 88026
2021-04-17 22:51:25iritkatrielsetnosy: + iritkatriel
messages: + msg391317
2021-04-17 07:34:25chelmitesetmessages: + msg391273
2021-04-16 06:26:17rhettingersetnosy: + rhettinger
messages: + msg391168
2021-04-16 05:39:02chelmitecreate