Follow-up of the issue #30314. Copy of Serhiy Storchaka's msg294048:
--
test_tools is so slow because it proceeds *all* Python files. This controlled by the "cpu" resource (only 10 random files are chosen if it is disabled).

Maybe disable the "cpu" resource on slow buildbots?
--

Good point, I like the idea.

It seems like regrtest accepts -u all,-cpu option.

And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu".

This made tests faster by a third!

One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, otherwise coverage shifts and we can't rely on any coverage metrics per-PR to know if code is increasing or decreasing coverage. Maybe having a ``-u deterministic`` resource to take out the randomness for coverage runs but leave the randomness for buildbots? This might also require tweaking our Travis tests as they currently use the buildbot make rule (I think).

P.S. I bet there are also some multiprocessing tests that go a bit overboard that we could consider scaling back, e.g. the coverage run skips a bunch of tests because they seem to process the entire stdlib.

New changeset c53b13b270767948fddb58b287149c499f9a03c4 by Zachary Ware in branch 'master':
bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
https://github.com/python/cpython/commit/c53b13b270767948fddb58b287149c499f9a03c4

New changeset d125738e93a35125d0edbe8070dd47160bcda829 by Zachary Ware in branch '3.6':
[3.6] bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
https://github.com/python/cpython/commit/d125738e93a35125d0edbe8070dd47160bcda829

New changeset 58e5dd1a03b3ce421f07085f8cd6e8566a1ed197 by Zachary Ware in branch '3.5':
[3.5] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
https://github.com/python/cpython/commit/58e5dd1a03b3ce421f07085f8cd6e8566a1ed197

New changeset ddf8f141f0181f9cac05c87383ea60887acffaec by Zachary Ware in branch '2.7':
[2.7] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
https://github.com/python/cpython/commit/ddf8f141f0181f9cac05c87383ea60887acffaec

New changeset 42e3acda86829def9adc354fbee77597b849bf9e by Zachary Ware in branch 'master':
bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/42e3acda86829def9adc354fbee77597b849bf9e

New changeset e380c19b7e88d97d3497016f1065d6922415db1c by Zachary Ware in branch '3.6':
[3.6] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/e380c19b7e88d97d3497016f1065d6922415db1c

New changeset 91581d4225e91e581f44d60d0b87c80778c36d1d by Zachary Ware in branch '3.5':
[3.5] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/91581d4225e91e581f44d60d0b87c80778c36d1d

New changeset 3c7b67bb34bc537b6c0dd9c6e1e4a07dc227b467 by Zachary Ware in branch '2.7':
[2.7] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/3c7b67bb34bc537b6c0dd9c6e1e4a07dc227b467

Thanks for taking care of our CI, Zach ;-) Buildbots, Travis CI job and AppVeyor are now fixed if I understood correctly, so I close the issue.


Serhiy Storchaka: "And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu"."

Hum, I have no real opinion on this issue. I think that the first main issue was test_tools, and disabling cpu on our CI has fixed the issue. I'm not sure that we should go deeper. *If* you consider thta that test_datetime is too slow, yeah, maybe add also a requirement on the "cpu" resource. But I don't recall to have seen "test_datetime" in the final "top 10 slowest tests" (even if I didn't look recently).


Brett Cannon: "One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, (...)"

Using a fixed list of filenames would benefit to everyone, not only to the CI. But someone has to select these files :-)

Brett: "Maybe having a ``-u deterministic`` resource to take out the randomness for coverage runs but leave the randomness for buildbots?"

We may start by using a fixed random seed for our coverage tests? And make sure that each test file starts with the same seed? IMHO coverage is a different issue, so I suggest to open a new issue if you consider that it's worth it to enhance the coverage CI job ;-)