This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: No facility for test randomisation
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ezio.melotti, michael.foord, rbcollins, sbspider
Priority: normal Keywords:

Created on 2014-11-07 10:05 by rbcollins, last changed 2022-04-11 14:58 by admin.

Messages (6)
msg230778 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-11-07 10:05
Unittest doesn't support a test randomisation feature.

Such a feature should support:
 - passing in a seed (to allow reproducing the order for debugging)
 - preserving the suite hierarchy, to preserve class and module setUp performance optimisations
 - and randomising globally with cloned suite hierarchies, for more comprehensive randomisation
 - allowing some scopes to opt out of randomisation (where tests really have dependencies on execution order and thats what the test author wants)

From https://code.google.com/p/unittest-ext/issues/detail?id=6
msg230836 - (view) Author: (sbspider) * Date: 2014-11-08 05:42
To clarify - are you querying about 

a) Randomiser for tests, so that tests can have random variables
or 
b) That the order in which tests are called should have the option to be randomized

?
msg230837 - (view) Author: Robert Collins (rbcollins) * (Python committer) Date: 2014-11-08 05:47
b)
msg230938 - (view) Author: (sbspider) * Date: 2014-11-10 07:41
And what do you think would it be useful for? I mean, it shouldn't be too hard to implement, a couple of hours at most, but I can't seem to see what would be the purpose. However, if you think that there is enough of a use case behind it, then I would be happy to have a crack at it :).
msg230948 - (view) Author: Michael Foord (michael.foord) * (Python committer) Date: 2014-11-10 09:49
The point is that it is easy to have unintentional dependencies between tests. Test a sets up some state that test b relies on. This means that test b passes, so long as test a has already run. This is bad, tests should be isolated - it also means you can break test b when you change test a. Randomising test run order means you discover these unintentional dependencies earlier.

With test randomisation you ideally need the seed to be displayed as part of the test run, and you need to be able to run with a particular seed. This enables you to reproduce failures, or odd results, from any particular test run.
msg230999 - (view) Author: (sbspider) * Date: 2014-11-11 06:27
Right makes sense. I'll see what I can do.
History
Date User Action Args
2022-04-11 14:58:09adminsetgithub: 67002
2015-03-02 08:40:31ezio.melottisettype: enhancement
stage: needs patch
2014-11-11 06:27:35sbspidersetmessages: + msg230999
2014-11-10 09:49:59michael.foordsetmessages: + msg230948
2014-11-10 07:41:35sbspidersetmessages: + msg230938
2014-11-08 05:47:38rbcollinssetmessages: + msg230837
2014-11-08 05:42:12sbspidersetnosy: + sbspider
messages: + msg230836
2014-11-07 22:44:28terry.reedysetnosy: + ezio.melotti, michael.foord
2014-11-07 10:05:59rbcollinscreate