Author slattarini
Recipients gregory.p.smith, michael.foord, slattarini
Date 2014-01-09.12:52:30
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1389271952.33.0.914436814798.issue20165@psf.upfronthosting.co.za>
In-reply-to
Content
Since I too was bitten by this issue, I'd like to support Gregory's
request, and report my rationale for changing the current behaviour.

With the current behaviour, we could see (and I *have* seen) scenarios
like this:
	
  1. A test exposing a known bug is written, and the test is marked
     as "expected failure".
	
  2. Some refactoring and code improvements follow.
	
  3. They improve the overall behaviour/correctness of the program
     or library under test, to the point of fixing the bug "behind
     the scenes".
	
  4. The developer doesn't notice the fix though, since the testing
     framework doesn't warn him "strongly enough" of the fact that the
     test marked as expected failure has begun to pass. (To reiterate,
     the current behaviour of just printing a warning saying "some test
     passed unexpectedly" on the standard output is not good enough of
     a warning; it's easy to miss, and worse, it's *certain* that it
     will be missed if the tests are run by some CI systems or a similar
     wrapper system -- those would only report failures due to non-zero
     exit statuses.)
	  
  5. Without noticing that the issue has been fixed, the developer does
     some further changes, which again re-introduce the bug, or a
     similar one that the test still marked as "expected failure" could
     easily catch.
	  
  6. That test starts to fail again; but since it has remained marked
     as "expected failure" all along, the fact is not reported to the
     developer in any way.  So the bug has managed to sneak back in,
     unnoticed.

In addition to this rationale, another (weaker) reason to change the
existing behaviour would be the "principle of least surprise".  Among
other widely used testing framework (for python, or language-agnostic)
many of those which support the concept of "expected failure" will
throw hard errors by default when a test marked as expected failure
starts to pass; among these are:

  * Python libs:
    - py.test (http://pytest.org)
    - Nose (https://pypi.python.org/pypi/nose/1.3.0)

  * Language-agnostic protocols/frameworks:
    - the TAP protocol (the standard in the Perl world)
      (http://en.wikipedia.org/wiki/Test_Anything_Protocol)
    - the Automake "Simple Tests" framework
      (http://www.gnu.org/software/automake/manual/html_node/Tests.html)
History
Date User Action Args
2014-01-09 12:52:32slattarinisetrecipients: + slattarini, gregory.p.smith, michael.foord
2014-01-09 12:52:32slattarinisetmessageid: <1389271952.33.0.914436814798.issue20165@psf.upfronthosting.co.za>
2014-01-09 12:52:32slattarinilinkissue20165 messages
2014-01-09 12:52:30slattarinicreate