classification
Title: Unittest runner needs an option to call gc.collect() after each test
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, azsorkin, ezio.melotti, giampaolo.rodola, gvanrossum, michael.foord, pconnell, pitrou, rbcollins, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2013-05-05 02:20 by gvanrossum, last changed 2019-07-29 11:47 by vstinner.

Files
File name Uploaded Description Edit
unittest-add-gc.patch azsorkin, 2015-10-07 15:09 Patch that adds command line option for garbage collection review
Messages (20)
msg188420 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-05 02:20
It would be nice if there was a command-line option to tell the test runner to call gc.collect() after each test.

See https://groups.google.com/d/msg/python-tulip/tstjvOQowIU/IRihc8NCHZUJ -- Glyph points out that always doing this is problematic because it slows down tests too much; OTOH it's a useful option to have in case you're tracking down something that happens (or doesn't happen) when an object is collected (e.g. in __del__).
msg188422 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2013-05-05 02:30
See also #17534.
msg188436 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-05 11:29
> OTOH it's a useful option to have in case you're tracking down
> something that happens (or doesn't happen) when an object is collected

IMO this is a good reason to implement your specific tearDown method (or call addCleanup if you prefer), possibly in a shared base class.

I don't think this is a good candidate for a command-line option, it's too specialized and it's also not something which should be enabled blindly. In other words, each test case should know whether it needs a collection or not.
msg188451 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-05 17:23
On Sun, May 5, 2013 at 4:29 AM, Antoine Pitrou <report@bugs.python.org> wrote:
>
> Antoine Pitrou added the comment:
>
>> OTOH it's a useful option to have in case you're tracking down
>> something that happens (or doesn't happen) when an object is collected
>
> IMO this is a good reason to implement your specific tearDown method (or call addCleanup if you prefer), possibly in a shared base class.
>
> I don't think this is a good candidate for a command-line option, it's too specialized and it's also not something which should be enabled blindly. In other words, each test case should know whether it needs a collection or not.

This is not for tests that know or expect they need a call to
gc.collect(). This is for the case where you have 500 tests that
weren't written with gc.collect() in mind, and suddenly you have a
nondeterministic failure because something goes wrong during
collection. The cause is probably many tests earlier -- and if you
could just call gc.collect() in each tearDown() it would be a cinch to
pinpoint the test that causes this. But (unless you had all that
foresight) that's a massive undertaking. However turning on the
command line option makes it trivial.

It's rare that extra collections cause tests to fail (and if it does
that's a flakey test anyway) so just turning this on for all tests
shouldn't affect correct tests -- however it can slow down your test
suite 3x or more so you don't want this to be unittest's default
behavior. Hence the suggestion of a command line flag.
msg188744 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-05-08 22:45
I'm afraid I'm inclined to agree with Antoine. This seems a relatively specialised problem and I'd expect to see it solved in the test system rather than in unittest itself.

One solution, and something I'd like to see, is a way for test systems to extend the unittest command line handling so that they can add extra options. At the moment it's very painful to do.
msg188745 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2013-05-08 22:53
TBH I would *love* for there to be a standardized way to extend the command line options!  If it existed I would help myself.  As it is, that's way to complicated.  (Then again, most customizations to the default test runner are too complicated IMO. :-)
msg188793 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2013-05-09 22:37
The default test runner is quite horrible.
msg188818 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-05-10 08:41
> TBH I would *love* for there to be a standardized way to extend the
> command line options!  If it existed I would help myself.  As it is,
> that's way to complicated.  (Then again, most customizations to the
> default test runner are too complicated IMO. :-)

+1. I found myself wanting to add a "-F" option à la regrtest, and I
ended up processing sys.argv manually :-/
msg242165 - (view) Author: Adam (azsorkin) * Date: 2015-04-28 05:03
Is this enhancement still open? I've run into this problem previously, and would be more than happy to implement this feature.
msg242166 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-04-28 05:19
Does the cpython test runner have this? Might be nice there. Not sure if
the default test runner is something to extend at this point. Eveybody is
using py.test anyway...

On Monday, April 27, 2015, Ned Deily <report@bugs.python.org> wrote:

>
> Changes by Ned Deily <nad@acm.org <javascript:;>>:
>
>
> ----------
> nosy: +rbcollins
>
> _______________________________________
> Python tracker <report@bugs.python.org <javascript:;>>
> <http://bugs.python.org/issue17908>
> _______________________________________
>
msg242168 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 07:33
I still think that it's something that people can trivially implement and that has no place in the standard unittest package.
msg242190 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-04-28 16:53
It's trivial to add to a single test or even a single TestCase subclass, yes. The use case is more that you have a ton of tests and you suspect there's a problem due to GC. Being able to call gc.collect() after each test through the flip of a flag would be useful.

But I agree it's iffy whether this belongs in the unittest package; we could add it to the CPython run_tests.py script, or you could request this as a feature from py.test.

If someone wants to add this to run_tests.py I think this issue would be the place to review the patch.
msg242191 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-28 17:30
See also issue23839 where proposed not only call gc.collect(), but clear all caches in test.regrtest.
msg242202 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-04-28 19:00
I'm going to disagree with michael and antoine here.

The *internals* should be clean and pluggable for sure, but this is actually a pretty common thing to try, so there's no reason to force it to only be done by external plugins.

Right now the way to plug this in has been complicated by the addition of module / class suites, which already perform extra work around individual tests, but in a non-introspectable / extensible fashion.

So you could add this as a hook to the loader (decorate each test with some new thing) and a CLI option to use that hook for a gc collect call.

Alternatively, we could face down the class/module stuff and rearrange it to be extensible (e.g. something along the lines of testresources internals - generic groups pre-post stuff) or via some interaction with TestResult.... but I really dislike using TestResult to control the run - I have a better layout mostly sketched in mind but haven't had time to formalise it.

So I recommend the TestLoader hook point - its well within the current responsibilities for it to do this, and I don't see any long term maintenance issues.
msg242220 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-29 11:04
Le 28/04/2015 21:00, Robert Collins a écrit :
> So you could add this as a hook to the loader (decorate each test
> with some new thing) and a CLI option to use that hook for a gc
> collect call.

Can I take this as meaning you're not opposed to adding other options to
the unittest core? (say, something to run a test until failure, or to
watch for reference leaks, or to run tests in multiple processes :-))
msg249033 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2015-08-23 23:59
"say, something to run a test until failure, or to
watch for reference leaks, or to run tests in multiple processes :-))"

I think a few complimentary things.

unittest extensability currently requires a new CLI entry point for each thing. I'd like to fix that.

The actual plumbing is fairly extensible, though its not always obvious. I'd like to fix that too - without any global state getting involved.

Of the things you mention, running a given command line until failure and checking for reference leaks are both straight forward, very common requests (as is the gc check) and I'd like to see those implemented as extensions shipped in the stdlib.

Running in parallel becomes important when one is doing slow (e.g. functional) tests with unittest, and I think thats important to support. It is however much harder to do well: some of the current idioms that have snuck in (like the handling of stdout/stderr capturing without the buffer flag) are not well matched to the needs of reporting on concurrent tests to users. I'm not in any way opposed to a good implementation, but it would need to be good I think - there's not much point having a poor implementation, given the rich set of parallel test runners that are out there that already build on the unittest core (green, nose, testrepository just for starters). The only unique audience for stdlib test facilities is the stdlib itself and I think a better way to solve that is to enable the use of alternative runners for our own tests (moving them to be a little cleaner should enable that) - and then the point where it matters is really 'when buildbots would be enough faster to make a difference'.
msg252957 - (view) Author: Adam (azsorkin) * Date: 2015-10-13 19:51
Any comments about this proposed patch?
msg252958 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-10-13 20:03
While I like the idea of adding an option to force a garbage collection after running each time, I don't like how unittest is growing :-( It looks more and more like regrtest and I hate regrtest (I reworked its code recently to create a less ugly "libregrtes").

Once Antoine Pitrou proposed to add "plugins" to unittest. I agree that we need a gate between unittest and all other test frameworks: nose, py.test, testtools/testrepository/testr, etc.

Seriously, don't you think that something is wrong with such API?

def __init__(self, module='__main__', defaultTest=None, argv=None,
             testRunner=None, testLoader=loader.defaultTestLoader,
             exit=True, verbosity=1, failfast=None, catchbreak=None,
             buffer=None, warnings=None, *, tb_locals=False,
             gc_enabled=False):
msg252962 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-10-13 20:59
Well, maybe we can add yet another parameter to all unittest classes
(push this change), but IMHO someone must rework the unittest API to
make it more extensible instead of adding more and more stuff in the
base API.
msg348635 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-07-29 11:47
This issue is no newcomer friendly, I remove the "easy" keyword.
History
Date User Action Args
2019-07-29 11:47:10vstinnersetkeywords: - easy

messages: + msg348635
versions: + Python 3.9, - Python 3.5
2015-10-13 20:59:59vstinnersetmessages: + msg252962
2015-10-13 20:03:52vstinnersetmessages: + msg252958
2015-10-13 19:51:14azsorkinsetmessages: + msg252957
2015-10-09 20:37:41@nkitsetnosy: - @nkit
2015-10-09 20:22:39@nkitsetnosy: + @nkit
2015-10-09 00:46:08rhettingersetnosy: + vstinner
2015-10-07 15:09:23azsorkinsetfiles: + unittest-add-gc.patch
keywords: + patch
2015-08-23 23:59:33rbcollinssetmessages: + msg249033
2015-04-29 11:04:45pitrousetmessages: + msg242220
2015-04-28 19:00:48rbcollinssetmessages: + msg242202
2015-04-28 17:30:40serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg242191
2015-04-28 16:53:17gvanrossumsetmessages: + msg242190
2015-04-28 07:33:18pitrousetmessages: + msg242168
versions: + Python 3.5, - Python 3.4
2015-04-28 05:20:00gvanrossumsetmessages: + msg242166
title: Unittest runner needs an option to call gc.collect() after each test -> Unittest runner needs an option to call gc.collect() after each test
2015-04-28 05:07:44ned.deilysetnosy: + rbcollins
2015-04-28 05:03:47azsorkinsetnosy: + azsorkin
messages: + msg242165
2015-02-13 01:23:11demian.brechtsetnosy: - demian.brecht
2013-08-27 09:46:28asvetlovsetnosy: + asvetlov
2013-06-29 19:27:32demian.brechtsetnosy: + demian.brecht
2013-05-11 06:38:18pconnellsetnosy: + pconnell
2013-05-10 08:41:42pitrousetmessages: + msg188818
title: Unittest runner needs an option to call gc.collect() after each test -> Unittest runner needs an option to call gc.collect() after each test
2013-05-09 22:37:42michael.foordsetmessages: + msg188793
2013-05-08 22:53:02gvanrossumsetmessages: + msg188745
2013-05-08 22:45:51michael.foordsetmessages: + msg188744
2013-05-05 17:23:55gvanrossumsetmessages: + msg188451
2013-05-05 11:29:55pitrousetnosy: + pitrou
messages: + msg188436
2013-05-05 11:21:58giampaolo.rodolasetnosy: + giampaolo.rodola
2013-05-05 02:30:16ezio.melottisetnosy: + ezio.melotti, michael.foord

messages: + msg188422
stage: needs patch
2013-05-05 02:20:44gvanrossumcreate