classification
Title: Find a way to detect incorrectly skipped tests
Type: enhancement Stage:
Components: Tests Versions:
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, ncoghlan, terry.reedy, xdegaye
Priority: normal Keywords:

Created on 2013-09-08 03:05 by ncoghlan, last changed 2015-07-21 08:11 by ethan.furman.

Messages (4)
msg197218 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-09-08 03:05
Issue 18952 (fixed in http://hg.python.org/cpython/rev/23770d446c73) was another case where a test suite change resulted in tests not be executed as expected, but this wasn't initially noticed since it didn't *fail* the tests, it just silently skipped them.

We've had similar issues in the past, due to test name conflicts (so the second test shadowed the first), to old regrtest style test discovery missing a class name from the test list, and to incorrect skip conditions on platform specific tests.

Converting "unexpected skips" to a failure isn't enough, since these errors occur at a narrower scope than entire test modules.

I'm not sure on what *would* work, though. Perhaps collecting platform specific coverage stats for the test suite itself and looking for regressions?
msg197228 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2013-09-08 06:55
Run the test suite both with and without the patch, and compare the results.  Additional skipped tests, additional failed tests, or less than the number of expected additional tests signal a problem.

The first two should be automatable, the last depends on the human paying attention.  ;)

This should at least deal with problems created by a patch.
msg197646 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-13 20:34
Isn't this a duplicate of #16056?
msg197647 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-09-13 20:35
See also 16079.
History
Date User Action Args
2015-07-21 08:11:36ethan.furmansetnosy: - ethan.furman
2013-09-14 07:04:30xdegayesetnosy: + xdegaye
2013-09-13 20:35:22ezio.melottisetmessages: + msg197647
2013-09-13 20:34:20ezio.melottisetnosy: + ezio.melotti
messages: + msg197646
2013-09-13 18:28:14terry.reedysetnosy: + terry.reedy
2013-09-13 18:28:06terry.reedysetcomponents: + Tests
2013-09-08 06:55:01ethan.furmansetnosy: + ethan.furman
messages: + msg197228
2013-09-08 03:05:57ncoghlansettitle: Find a way to detect regressions in test execution -> Find a way to detect incorrectly skipped tests
2013-09-08 03:05:34ncoghlancreate