classification
Title: Documentation of TestCase.runTest is incorrect and confusing
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.6, Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: berker.peksag, docs@python, evilzero, ezio.melotti, martin.panter, michael.foord, pitrou, python-dev, rbcollins
Priority: normal Keywords: patch

Created on 2014-08-06 05:00 by martin.panter, last changed 2015-07-22 19:25 by rbcollins. This issue is now closed.

Files
File name Uploaded Description Edit
myworkdoc.patch evilzero, 2014-08-17 20:00 review
runtest.patch martin.panter, 2014-12-05 23:53 review
runTest2-default.patch martin.panter, 2014-12-06 23:58 review
runTest2-3.4.patch martin.panter, 2014-12-06 23:58 review
Messages (17)
msg224907 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-08-06 05:00
The documentation for "unittest.TestCase" says "the standard implementation of the default 'methodName', runTest(), will run every method starting with 'test' as an individual test". However:

>>> from unittest import *
>>> class Test(TestCase):
...     def test_method(self): pass
... 
>>> t = Test()
>>> t.run()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/unittest/case.py", line 552, in run
    testMethod = getattr(self, self._testMethodName)
AttributeError: 'Test' object has no attribute 'runTest'

After further experimentation, I see that if my test method is called "runTest", it can be automatically discovered, but only if there are no other test- prefixed methods.

Perhaps you could drop the end of the second paragraph for TestCase, so that it just reads:

Each instance of TestCase will run a single base method: the method named "methodName".

I think the details about the test- prefix and counting results are covered elsewhere, and in most uses you wouldn't instantiate a TestCase yourself, so changing the method name is irrelevant.

Also, perhaps under "TestLoader.loadTestsFromTestCase" it should say:

If no methods with the usual name prefix are found, but the "runTest" method is implemented, there will be a single test case using that method.
msg225466 - (view) Author: (evilzero) * Date: 2014-08-17 20:00
Updated documentation following suggestion.
msg230137 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-10-28 10:52
The patch seems reasonable to me
msg230147 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-28 13:16
IMO hiding the existence of `runTest` would be best. It doesn't seem to make anything more flexible, and it complicates the documentation.
msg230180 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-10-28 21:57
Do you mean pretending there is no default “methodName” value, or pretending that the runTest() method is not invoked by discovery?

I would have to check, but I think I have relied on the runTest() method being discovered in the past, when I did not think a more original test method name was useful. Though I agree that it makes the behaviour more complicated for little extra flexibility.
msg230181 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-10-28 22:01
runTest is part of the current API. I think if we're going to hide it we should do so as part of a deprecation path. (I'm +1 on hiding it).
msg230182 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-28 22:06
Le 28/10/2014 23:01, Robert Collins a écrit :
> 
> runTest is part of the current API. I think if we're going to hide
> it
we should do so as part of a deprecation path. (I'm +1 on hiding it).

We don't need a deprecation path to remove something from the
documentation, IMO. It wouldn't break any existing code, it would just
reduce the cognitive load for newcomers.

(and, while people may have chosen to use runTest, it doesn't mean they
need it in any way)
msg230188 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-10-28 23:22
Removing stuff from the documentation would make it hard to understand what existing code means that uses runTest(). Isn’t this a case where you would add a big red “Deprecated since version . . .” warning instead? Maybe a comprimise would be to “hide” its documentation away in a separate deprecated section or something.
msg230190 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-29 00:16
Le 29/10/2014 00:22, Martin Panter a écrit :
> 
> Martin Panter added the comment:
> 
> Removing stuff from the documentation would make it hard to
> understand
what existing code means that uses runTest(). Isn’t this a case where
you would add a big red “Deprecated since version . . .” warning
instead? Maybe a comprimise would be to “hide” its documentation away in
a separate deprecated section or something.

Well, I wasn't proposing a deprecation (although that can be done as
well, if we want). I was proposing that this detail be hidden. I don't
know if it would make existing software harder to understand. I don't
think I've ever seen an actively-maintained project that used runTest().

The simple way to hide it would be to stop documenting the TestCase
constructor and its signature. You aren't supposed to instantiate
TestCase yourself.
msg230191 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-10-29 00:25
Constructing test case objects directly is part of the protocol, and its needed for framework and extension authors.

I've seen folk use runTest, but rarely enough that I think we could deprecate it. My argument is that while its supported we should be clear about when it should be used, and how.
msg230231 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-10-29 18:50
Oh, one thought - in testtools we split out 'docs for test writers' and 'docs for framework folk'. That would facilitate getting runTest out of test writers face while still documenting it appropriately.

Related to that is my ongoing push to split the protocol so that these concerns are not intertwined the way they are today, but that needs to wait for the existing backlog to clear up :)
msg230312 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-10-30 23:53
I'll apply the patch monday if there are no further comments before then.
msg230352 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2014-10-31 16:55
> Oh, one thought - in testtools we split out 'docs for test writers' and
> 'docs for framework folk'. That would facilitate getting runTest out of
> test writers face while still documenting it appropriately.

This has already been discussed in the past, and IIRC there was consensus about giving unittest docs the same treatment.  I don't remember if there was an issue for this.
msg232226 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-05 23:53
Updated patch with indentation fixed and new wording. I am just guessing the RST syntax based on the surrounding text, so please review :)
msg232260 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2014-12-06 23:58
Updated patch, which applies to current tip of the default branch, and includes the formatting fix. Also including a version that applies to the 3.4 branch. Alternatively, if you patch the 3.4 branch it looks like merging to default automatically gives the correct result.
msg247143 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-07-22 18:40
New changeset eefc157b3096 by Robert Collins in branch '3.4':
Issue #22153: Improve unittest docs. Patch from Martin Panter and evilzero.
https://hg.python.org/cpython/rev/eefc157b3096

New changeset 10f5a7fa26d5 by Robert Collins in branch '3.5':
Issue #22153: Improve unittest docs. Patch from Martin Panter and evilzero.
https://hg.python.org/cpython/rev/10f5a7fa26d5

New changeset 45bd2dadbd0d by Robert Collins in branch 'default':
Issue #22153: Improve unittest docs. Patch from Martin Panter and evilzero.
https://hg.python.org/cpython/rev/45bd2dadbd0d
msg247145 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-07-22 18:43
Thanks for the update, it looks good to me. Applied to 3.4 and up. I'm not applying to 2.7 at this stage.
History
Date User Action Args
2015-07-22 19:25:05rbcollinssetstatus: open -> closed
stage: commit review -> resolved
resolution: fixed
versions: + Python 3.6, - Python 2.7
2015-07-22 18:43:08rbcollinssetmessages: + msg247145
2015-07-22 18:40:40python-devsetnosy: + python-dev
messages: + msg247143
2014-12-06 23:58:33martin.pantersetfiles: + runTest2-3.4.patch
2014-12-06 23:58:20martin.pantersetfiles: + runTest2-default.patch

messages: + msg232260
2014-12-06 16:31:42berker.peksagsetnosy: + berker.peksag

stage: patch review -> commit review
2014-12-05 23:53:07martin.pantersetfiles: + runtest.patch

messages: + msg232226
2014-10-31 16:55:48ezio.melottisetmessages: + msg230352
2014-10-30 23:53:52rbcollinssetmessages: + msg230312
2014-10-29 19:00:06rbcollinssettitle: There is no standard TestCase.runTest implementation -> Documentation of TestCase.runTest is incorrect and confusing
2014-10-29 18:50:59rbcollinssetmessages: + msg230231
2014-10-29 00:25:21rbcollinssetmessages: + msg230191
2014-10-29 00:16:45pitrousetmessages: + msg230190
2014-10-28 23:22:50martin.pantersetmessages: + msg230188
2014-10-28 22:06:08pitrousetmessages: + msg230182
2014-10-28 22:01:05rbcollinssetmessages: + msg230181
2014-10-28 21:57:20martin.pantersetmessages: + msg230180
2014-10-28 13:16:18pitrousetnosy: + pitrou
messages: + msg230147
2014-10-28 10:52:31martin.pantersetmessages: + msg230137
2014-10-28 09:14:04ezio.melottisetnosy: + rbcollins
stage: patch review

versions: + Python 2.7
2014-08-17 20:01:10evilzerosettype: enhancement
2014-08-17 20:00:18evilzerosetfiles: + myworkdoc.patch

nosy: + evilzero
messages: + msg225466

keywords: + patch
2014-08-08 19:38:33terry.reedysetnosy: + ezio.melotti, michael.foord
2014-08-06 05:00:02martin.pantercreate