classification
Title: doctest.DocTestCase fails when run repeatedly
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, jamesh, mgedmin, pjd
Priority: normal Keywords: patch

Created on 2008-04-10 03:40 by pjd, last changed 2012-08-08 16:39 by mgedmin.

Files
File name Uploaded Description Edit
reset_globs.patch pjd, 2008-04-19 06:23 Zope's fix, plus a test docstring tweak review
Messages (8)
msg65285 - (view) Author: Piet Delport (pjd) Date: 2008-04-10 03:40
DocTestCase.tearDown destructively clears its DocTest instance's globs,
preventing the test from being run repeatedly (such as with trial
--until-failure).

There's a fix for this in zope.testing's version of doctest, which
resets the globs instead:

http://svn.zope.org/?view=rev&rev=39023
msg65286 - (view) Author: Piet Delport (pjd) Date: 2008-04-10 04:23
Addendum:  This appears to be a regression in r36809 (Python 2.4+).
msg65335 - (view) Author: James Henstridge (jamesh) Date: 2008-04-11 06:26
Is repeating a test with the same TestCase instance ever safe?  It'd be
better to create a new instance and run that.

If any of the variables in test.globs are changed by the test (e.g.
appending to a list), then rerunning the test will not necessarily give
the same result.

While the zope change allows tests that have immutable globals or don't
change their globals to function, it also lets other tests "almost"
work, and could lead to new bugs that are difficult to track down.
msg65339 - (view) Author: Piet Delport (pjd) Date: 2008-04-11 08:00
> If any of the variables in test.globs are changed by the test (e.g. 
appending to a list), then rerunning the test will not necessarily give
the same result.

This is true, but modifying the globals such that subsequent runs of the
same test can break equally affects subsequent runs of any other tests
that use that module:  such a test is already broken (unsafe to run with
other tests) to begin with, independent of the DocTestCase.tearDown issue.
msg65344 - (view) Author: James Henstridge (jamesh) Date: 2008-04-11 09:22
If I create a test case with a command like:

   test = DocFileSuite('foo.txt', globs={'somelist': [42]})

The doctest isn't doing anything wrong if it modifies somelist.

Furthermore, Glyph has said he thinks the current --until-failure
behaviour in trial is a mistake: http://glyf.livejournal.com/72505.html
msg65345 - (view) Author: Piet Delport (pjd) Date: 2008-04-11 10:29
Well, whether that code is wrong depends on whether your project policy
wants repeatable tests or not.  A repeatable and arguably more idiomatic
way of writing that example is to give DocFileSuite a setUp function
which initializes any special globals required by the test.

In any case, DocTestCase allowing non-repeatable tests (which i don't
think are common) is no reason to disallow repeatable tests, which e.g.
at least Zope considers important enough to motivate fixing this issue.
 (Zope is also the source of the code being fixed, if i'm not mistaken.)
msg65346 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-04-11 11:13
FWIW, the python testsuite needs repeatable tests, when running in
"reference leaks" mode.

See also r62100, where a DocTestSuite construction had to be moved into
the repeated function.
msg167696 - (view) Author: Marius Gedminas (mgedmin) * Date: 2012-08-08 16:39
For the record, this bug also breaks zope.testrunner's --repeat option, if you have any doctests in your test suite that rely on test.globs not going away.
History
Date User Action Args
2016-05-28 20:44:12berker.peksaglinkissue9327 superseder
2012-08-08 16:39:48mgedminsetnosy: + mgedmin
messages: + msg167696
2010-06-09 22:12:13terry.reedysetversions: + Python 3.1, Python 2.7, Python 3.2, - Python 2.6, Python 2.5, Python 2.4, Python 3.0
2008-04-19 06:23:33pjdsetfiles: + reset_globs.patch
keywords: + patch
2008-04-11 11:13:43amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg65346
2008-04-11 10:29:03pjdsetmessages: + msg65345
2008-04-11 09:22:31jameshsetmessages: + msg65344
2008-04-11 08:00:02pjdsetmessages: + msg65339
2008-04-11 06:26:21jameshsetnosy: + jamesh
messages: + msg65335
2008-04-10 04:23:42pjdsetmessages: + msg65286
2008-04-10 03:40:45pjdcreate