3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses #65311

wkschwartz · 2014-03-31T14:20:27Z

BPO	21112
Nosy	@pitrou, @larryhastings, @rbtcollins, @ned-deily, @ezio-melotti, @voidspace, @berkerpeksag, @wkschwartz
Files	test_expectedFailure.py: Test case for unittest.expecteFailure regression issue21112.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/berkerpeksag'
closed_at = <Date 2015-08-27.22:38:35.450>
created_at = <Date 2014-03-31.14:20:26.695>
labels = ['tests', 'type-bug', 'library']
title = '3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses'
updated_at = <Date 2015-08-27.22:38:35.449>
user = 'https://github.com/wkschwartz'

bugs.python.org fields:

activity = <Date 2015-08-27.22:38:35.449>
actor = 'rbcollins'
assignee = 'berker.peksag'
closed = True
closed_date = <Date 2015-08-27.22:38:35.450>
closer = 'rbcollins'
components = ['Library (Lib)', 'Tests']
creation = <Date 2014-03-31.14:20:26.695>
creator = 'William.Schwartz'
dependencies = []
files = ['34680', '34681']
hgrepos = []
issue_num = 21112
keywords = ['patch', '3.4regression']
message_count = 16.0
messages = ['215240', '218491', '218492', '218683', '218727', '218731', '226552', '226553', '226562', '229543', '229545', '229589', '238091', '238168', '249266', '249267']
nosy_count = 9.0
nosy_names = ['pitrou', 'larry', 'rbcollins', 'ned.deily', 'ezio.melotti', 'michael.foord', 'python-dev', 'berker.peksag', 'William.Schwartz']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21112'
versions = ['Python 3.4', 'Python 3.5']

wkschwartz · 2014-03-31T14:20:26Z

In Python 2.7 and 3.3, decorating a unittest.TestCase subclass with unittest.expectedFailure caused test discover to skip the decorated test case. Python 3.4 apparently ignores the decorator when applied to classes.

The attached file when run with Python 2.7.6 on Mac OS X produces the following output

$ python -m unittest discover
F.

======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

Ran 2 tests in 0.000s

FAILED (failures=1)

When run with Python 3.4.0 produces the following output.

~/Documents/testtest $ python3 -m unittest discover
F.F.
======================================================================
FAIL: test_fails (test.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

======================================================================
FAIL: test_fails (test.TestTreatment)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/schwartz_w/Documents/testtest/test.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

Ran 4 tests in 0.001s

FAILED (failures=2)

The expectedFailure decorator when applied to a class should either skip the class or run all of its tests and check that they failed for consistency with how expectedFailure applies to test methods.

berkerpeksag · 2014-05-13T23:20:07Z

Is there a chance to get this into 3.4.1?

ned-deily · 2014-05-13T23:22:47Z

Berker, if you think it should go into 3.4.1, you need to set the priority to "release blocker" to bring it to the attention of the release manager.

larryhastings · 2014-05-16T21:05:43Z

Berker: do you consider your diff ready to go in, or is it an "early" diff (like a work-in-progress)?

berkerpeksag · 2014-05-18T01:37:10Z

Berker: do you consider your diff ready to go in, or is it an "early"
diff (like a work-in-progress)?

I tested my patch with test_expectedFailure.py again. The patch is not really fixes the problem described in msg215240. So, I consider it a WIP patch for now. Sorry for the noise.

Here is the output on Python 3.3:

$ python3.3 -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

Ran 2 tests in 0.001s

FAILED (failures=1)

And here is the output on Python 3.5 (with bpo-21112.diff patch):

$ ./../python -m unittest discover -v
test_fails (test_expectedFailure.TestControl) ... FAIL
test_passes (test_expectedFailure.TestControl) ... ok
test_fails (test_expectedFailure.TestTreatment) ... expected failure
test_passes (test_expectedFailure.TestTreatment) ... unexpected success

======================================================================
FAIL: test_fails (test_expectedFailure.TestControl)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/berker/projects/cpython-default/playground/test_expectedFailure.py", line 9, in test_fails
    self.assertFalse(True)
AssertionError: True is not false

Ran 4 tests in 0.002s

FAILED (failures=1, expected failures=1, unexpected successes=1)

larryhastings · 2014-05-18T03:24:56Z

Considering that I'm tagging 3.4.1 within an hour or two, and we don't have a patch yet, I'd say that this is too late to go into 3.4.1. But I'm happy to consider it for a future 3.4.x revision.

ned-deily · 2014-09-08T02:04:04Z

Should this be fixed for 3.4.2?

larryhastings · 2014-09-08T02:05:10Z

Note: current plan for 3.4.2 is to release at the end of the month. RC1 will be in about a week.

voidspace · 2014-09-08T07:42:44Z

The patch looks good to me.

wkschwartz · 2014-10-16T18:06:47Z

3.4.2 has been released, it seems, without this getting fixed.

3.4.3 then? Are we still happy with the patch?

berkerpeksag · 2014-10-16T18:20:53Z

3.4.3 then? Are we still happy with the patch?

Please see msg218727. The patch is not ready yet.

pitrou · 2014-10-17T17:59:38Z

Berker, it looks like the 3.3 behaviour was buggy: TestTreatment isn't run at all! Instead, it should be run and consider failures as success.

So your patch appears more correct than what 3.3 did.

wkschwartz · 2015-03-14T17:37:09Z

3.4.3 has been released, it seems, without this getting fixed.

3.4.4 then?

--
William Schwartz

On Mon, Sep 8, 2014 at 3:42 AM, Michael Foord <report@bugs.python.org>
wrote:

Michael Foord added the comment:

The patch looks good to me.

----------

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21112\>

rbtcollins · 2015-03-16T01:28:07Z

Test looks good to me. Do you want to apply it?

python-dev · 2015-08-27T22:36:36Z

New changeset a90c6d608b85 by Robert Collins in branch '3.4':
Issue bpo-21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/a90c6d608b85

New changeset ac3f1a6b1de2 by Robert Collins in branch '3.5':
Issue bpo-21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/ac3f1a6b1de2

New changeset bf789ae9bde7 by Robert Collins in branch 'default':
Issue bpo-21112: Fix regression in unittest.expectedFailure on subclasses.
https://hg.python.org/cpython/rev/bf789ae9bde7

rbtcollins · 2015-08-27T22:38:26Z

2.7 is sufficiently different that I haven't backported this there - plus the report said it was introduced in 3.4. If someone can show this doesn't work in 2.7 and also supply a patch, we could apply it there.

wkschwartz mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Mar 31, 2014

ezio-melotti added the tests Tests in the Lib/test dir label Mar 31, 2014

berkerpeksag self-assigned this Mar 16, 2015

rbtcollins closed this as completed Aug 27, 2015

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses #65311

3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses #65311

wkschwartz mannequin commented Mar 31, 2014

wkschwartz mannequin commented Mar 31, 2014

berkerpeksag commented May 13, 2014

ned-deily commented May 13, 2014

larryhastings commented May 16, 2014

berkerpeksag commented May 18, 2014

larryhastings commented May 18, 2014

ned-deily commented Sep 8, 2014

larryhastings commented Sep 8, 2014

voidspace commented Sep 8, 2014

wkschwartz mannequin commented Oct 16, 2014

berkerpeksag commented Oct 16, 2014

pitrou commented Oct 17, 2014

wkschwartz mannequin commented Mar 14, 2015

rbtcollins commented Mar 16, 2015

python-dev mannequin commented Aug 27, 2015

rbtcollins commented Aug 27, 2015

3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses #65311

3.4 regression: unittest.expectedFailure no longer works on TestCase subclasses #65311

Comments

wkschwartz mannequin commented Mar 31, 2014

wkschwartz mannequin commented Mar 31, 2014

berkerpeksag commented May 13, 2014

ned-deily commented May 13, 2014

larryhastings commented May 16, 2014

berkerpeksag commented May 18, 2014

larryhastings commented May 18, 2014

ned-deily commented Sep 8, 2014

larryhastings commented Sep 8, 2014

voidspace commented Sep 8, 2014

wkschwartz mannequin commented Oct 16, 2014

berkerpeksag commented Oct 16, 2014

pitrou commented Oct 17, 2014

wkschwartz mannequin commented Mar 14, 2015

rbtcollins commented Mar 16, 2015

python-dev mannequin commented Aug 27, 2015

rbtcollins commented Aug 27, 2015