classification
Title: Clarify unittest expectedFailure behaviour in the documentation
Type: behavior Stage: resolved
Components: Documentation, Tests Versions: Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Kit Choi2, docs@python, iritkatriel, miss-islington, steve.dower
Priority: normal Keywords: patch

Created on 2019-09-30 09:14 by Kit Choi2, last changed 2020-11-18 22:43 by iritkatriel. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 22740 merged iritkatriel, 2020-10-17 14:52
PR 22782 merged miss-islington, 2020-10-19 21:27
PR 22783 merged miss-islington, 2020-10-19 21:28
Messages (4)
msg353557 - (view) Author: Kit Choi (Kit Choi2) Date: 2019-09-30 09:14
Following discussion in https://bugs.python.org/issue38296

The docstring of unittest.expectedFailure is misleading for people who differentiate "error" and "failure" when they read the sentence. This has a consequence of developers using the decorator without noticing unexpected errors are also silenced, which mean running a risk of their tests becoming unmaintained and invalid in the future.

I suggest updating the documentation to include a mention of the current behaviour of silencing unexpected errors, so that developers are aware of this.

Something like this?:

Mark the test such that unexpected success results in a failure. If an exception (BaseException excluding KeyboardInterrupt) occurs, the test will be considered a success. If the test passes, it will be considered a failure.
msg379023 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2020-10-19 21:27
New changeset fa8748271a61177e9bf609921fa464cc6990478b by Irit Katriel in branch 'master':
bpo-38320: Clarify that expectedFailure is satisfied by either failure or error of the test. (GH-22740)
https://github.com/python/cpython/commit/fa8748271a61177e9bf609921fa464cc6990478b
msg381377 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 22:37
New changeset f9fa920c30326050a7096c5cb3594465d1e75ff2 by Miss Islington (bot) in branch '3.9':
[3.9] bpo-38320: Clarify that expectedFailure is satisfied by either failure or error of the test. (GH-22740) (GH-22783)
https://github.com/python/cpython/commit/f9fa920c30326050a7096c5cb3594465d1e75ff2
msg381378 - (view) Author: miss-islington (miss-islington) Date: 2020-11-18 22:39
New changeset 5f463e501b9667d1059a1e916d59d19cdd6addf7 by Miss Islington (bot) in branch '3.8':
[3.8] bpo-38320: Clarify that expectedFailure is satisfied by either failure or error of the test. (GH-22740) (GH-22782)
https://github.com/python/cpython/commit/5f463e501b9667d1059a1e916d59d19cdd6addf7
History
Date User Action Args
2020-11-18 22:43:58iritkatrielsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-11-18 22:39:50miss-islingtonsetmessages: + msg381378
2020-11-18 22:37:45miss-islingtonsetmessages: + msg381377
2020-10-19 21:28:13miss-islingtonsetpull_requests: + pull_request21742
2020-10-19 21:27:35miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request21741
2020-10-19 21:27:25steve.dowersetnosy: + steve.dower
messages: + msg379023
2020-10-17 14:52:57iritkatrielsetkeywords: + patch
nosy: + iritkatriel

pull_requests: + pull_request21703
stage: patch review
2019-09-30 09:14:01Kit Choi2create