Title: Run Python's slowest tests in the first 3/4 of tests when using -r
Type: Stage:
Components: Tests Versions: Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, serhiy.storchaka, terry.reedy
Priority: normal Keywords:

Created on 2017-05-28 19:41 by brett.cannon, last changed 2017-06-03 18:25 by brett.cannon.

Messages (5)
msg294658 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-05-28 19:41
If we guaranteed that the slowest tests in the test suite started early enough in the test run to complete before the final test finished, we could shave off several minutes worth of time in cases where the slowest tests are the hold-up. So perhaps if we randomly inserted the 10 slowest tests in the first 3/4 of the complete test list we could make sure they finish early enough?
msg294660 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-05-28 20:20
One way to control this would allow `-r` to take an optional `fast` argument to do this selective shuffle of the slowest tests, i.e. `-r fast` would still randomize but control for the slowest tests showing up in the first 75% of tests.
msg295061 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-03 00:25
Starting longer running tests sooner only make sense if one runs them in parallel with -jn.  I just tried installed 64 bit 3.6.1 with -r -j12 (forgot -ugui).  Total duration: 108 seconds.  The last to finish was test_multiprocessing_spawn after 84 seconds, 5 seconds after the 2nd last.  So it started at 24 seconds, when about 150 of 405 tests had finished.  To not be a blocker, it needed to start at about 19 seconds, when 120 tests had finished.

I re-ran with 32 bit repository debug build and was not so lucky.  Test 404 finished at 3:24.  Test_multiprocessing_spawn finished 1:33 later at 4:57, after 121 seconds.  It started at 2:57, when about 352 of 405 had finished.  Test 303 (3/4the) finished at 2:41, 16 seconds earlier.

Conclusion: the 3/4 restriction will help, but is insufficient to really balance process loads.
msg295090 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-03 17:18
See also issue30417.
msg295095 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-06-03 18:25
Travis takes nearly 22 minutes to do a test run with the slowest test currently at just under 8 minutes. That means on Travis the slowest test takes roughly 1/3 of the time. But with your machine taking literally 2 minutes, or 1/10 the time as Travis, to do a test run, Terry, I'm not worried about speeding up your machine. ;)

But yes, we could change it to being the first 50% if people wanted. The 75% was somewhat arbitrarily selected to not artificially constrain the introduced pseudo-randomness of the test order.
Date User Action Args
2017-06-03 18:25:10brett.cannonsetmessages: + msg295095
2017-06-03 17:18:24serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg295090
2017-06-03 00:25:16terry.reedysetnosy: + terry.reedy
messages: + msg295061
2017-05-28 20:20:05brett.cannonsetmessages: + msg294660
2017-05-28 19:41:20brett.cannoncreate