classification
Title: 3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses
Type: behavior Stage: resolved
Components: Library (Lib), Tests Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: berker.peksag Nosy List: William.Schwartz, berker.peksag, ezio.melotti, larry, michael.foord, ned.deily, pitrou, python-dev, rbcollins
Priority: normal Keywords: 3.4regression, patch

Created on 2014-03-31 14:20 by William.Schwartz, last changed 2015-08-27 22:38 by rbcollins. This issue is now closed.

Files
File name Uploaded Description Edit
test_expectedFailure.py William.Schwartz, 2014-03-31 14:20 Test case for unittest.expecteFailure regression
issue21112.diff berker.peksag, 2014-03-31 15:11 review
Messages (16)
msg215240 - (view) Author: William Schwartz (William.Schwartz) * Date: 2014-03-31 14:20
In Python 2.7 and 3.3, decorating a unittest.TestCase subclass with unittest.expectedFailure caused test discover to skip the decorated test case. Python 3.4 apparently ignores the decorator when applied to classes.

The attached file when run with Python 2.7.6 on Mac OS X produces the following output

$ python -m unittest discover
F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 0.000s

FAILED (failures=1)

When run with Python 3.4.0 produces the following output.

~/Documents/testtest $ python3 -m unittest discover
F.F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

======================================================================
FAIL: test_fails (test.TestTreatment)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)


The expectedFailure decorator when applied to a class should either skip the class or run all of its tests and check that they failed for consistency with how expectedFailure applies to test methods.
msg218491 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-05-13 23:20
Is there a chance to get this into 3.4.1?
msg218492 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-05-13 23:22
Berker, if you think it should go into 3.4.1, you need to set the priority to "release blocker" to bring it to the attention of the release manager.
msg218683 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-05-16 21:05
Berker: do you consider your diff ready to go in, or is it an "early" diff (like a work-in-progress)?
msg218727 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-05-18 01:37
> Berker: do you consider your diff ready to go in, or is it an "early"
> diff (like a work-in-progress)?

I tested my patch with test_expectedFailure.py again. The patch is not really fixes the problem described in msg215240. So, I consider it a WIP patch for now. Sorry for the noise.

Here is the output on Python 3.3:

$ python3.3 -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 2 tests in 0.001s

FAILED (failures=1)


And here is the output on Python 3.5 (with issue21112.diff patch):

$ ./../python -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok
test_fails (test_expectedFailure.TestTreatment) ... expected failure
test_passes (test_expectedFailure.TestTreatment) ... unexpected success

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

----------------------------------------------------------------------
Ran 4 tests in 0.002s

FAILED (failures=1, expected failures=1, unexpected successes=1)
msg218731 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-05-18 03:24
Considering that I'm tagging 3.4.1 within an hour or two, and we don't have a patch yet, I'd say that this is too late to go into 3.4.1.  But I'm happy to consider it for a future 3.4.x revision.
msg226552 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-09-08 02:04
Should this be fixed for 3.4.2?
msg226553 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-09-08 02:05
Note: current plan for 3.4.2 is to release at the end of the month.  RC1 will be in about a week.
msg226562 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-09-08 07:42
The patch looks good to me.
msg229543 - (view) Author: William Schwartz (William.Schwartz) * Date: 2014-10-16 18:06
3.4.2 has been released, it seems, without this getting fixed.

3.4.3 then? Are we still happy with the patch?
msg229545 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-10-16 18:20
> 3.4.3 then? Are we still happy with the patch?

Please see msg218727. The patch is not ready yet.
msg229589 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-17 17:59
Berker, it looks like the 3.3 behaviour was buggy: TestTreatment isn't run at all! Instead, it should be run and consider failures as success.

So your patch appears more correct than what 3.3 did.
msg238091 - (view) Author: William Schwartz (William.Schwartz) * Date: 2015-03-14 17:37
3.4.3 has been released, it seems, without this getting fixed.

3.4.4 then?

-- 
William Schwartz

On Mon, Sep 8, 2014 at 3:42 AM, Michael Foord <report@bugs.python.org>
wrote:

>
> Michael Foord added the comment:
>
> The patch looks good to me.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21112>
> _______________________________________
>
msg238168 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-03-16 01:28
Test looks good to me. Do you want to apply it?
msg249266 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-08-27 22:36
New changeset a90c6d608b85 by Robert Collins in branch '3.4':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/a90c6d608b85

New changeset ac3f1a6b1de2 by Robert Collins in branch '3.5':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/ac3f1a6b1de2

New changeset bf789ae9bde7 by Robert Collins in branch 'default':
Issue #21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/bf789ae9bde7
msg249267 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-27 22:38
2.7 is sufficiently different that I haven't backported this there - plus the report said it was introduced in 3.4. If someone can show this doesn't work in 2.7 and also supply a patch, we could apply it there.
History
Date User Action Args
2015-08-27 22:38:35rbcollinssetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2015-08-27 22:38:26rbcollinssetmessages: + msg249267
2015-08-27 22:36:36python-devsetnosy: + python-dev
messages: + msg249266
2015-08-26 00:45:30rbcollinssetstage: patch review -> commit review
2015-03-16 01:34:20berker.peksagsetassignee: berker.peksag
2015-03-16 01:28:07rbcollinssetmessages: + msg238168
2015-03-14 17:37:09William.Schwartzsetmessages: + msg238091
2014-10-17 17:59:37pitrousetmessages: + msg229589
2014-10-16 18:20:53berker.peksagsetnosy: + rbcollins
messages: + msg229545
2014-10-16 18:06:46William.Schwartzsetmessages: + msg229543
2014-09-08 07:42:44michael.foordsetmessages: + msg226562
2014-09-08 02:05:09larrysetmessages: + msg226553
2014-09-08 02:04:03ned.deilysetmessages: + msg226552
2014-05-18 03:24:56larrysetmessages: + msg218731
2014-05-18 01:37:10berker.peksagsetmessages: + msg218727
2014-05-16 21:05:43larrysetmessages: + msg218683
2014-05-16 20:08:49ned.deilysetnosy: + larry
2014-05-13 23:22:46ned.deilysetnosy: + ned.deily
messages: + msg218492
2014-05-13 23:20:07berker.peksagsetstage: patch review
messages: + msg218491
versions: + Python 3.5
2014-03-31 15:11:25berker.peksagsetfiles: + issue21112.diff
nosy: + berker.peksag
keywords: + patch
2014-03-31 14:22:51ezio.melottisetkeywords: + 3.4regression
nosy: + pitrou, ezio.melotti, michael.foord
components: + Tests
2014-03-31 14:20:26William.Schwartzcreate