classification
Title: [buildbot] Disable the cpu resources on slow buildbots
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: belopolsky, brett.cannon, haypo, pitrou, serhiy.storchaka, zach.ware
Priority: normal Keywords:

Created on 2017-05-20 22:06 by haypo, last changed 2017-06-10 09:06 by haypo. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 1928 merged zach.ware, 2017-06-03 16:41
PR 1951 merged zach.ware, 2017-06-04 22:33
PR 1961 merged zach.ware, 2017-06-05 23:50
PR 1962 merged zach.ware, 2017-06-05 23:51
PR 1964 merged zach.ware, 2017-06-06 00:01
PR 2057 merged zach.ware, 2017-06-10 01:24
PR 2058 merged zach.ware, 2017-06-10 01:25
PR 2059 merged zach.ware, 2017-06-10 01:30
Messages (13)
msg294052 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-05-20 22:06
Follow-up of the issue #30314. Copy of Serhiy Storchaka's msg294048:
--
test_tools is so slow because it proceeds *all* Python files. This controlled by the "cpu" resource (only 10 random files are chosen if it is disabled).

Maybe disable the "cpu" resource on slow buildbots?
--

Good point, I like the idea.

It seems like regrtest accepts -u all,-cpu option.
msg295086 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-03 16:37
And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu".
msg295089 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-06-03 17:17
This made tests faster by a third!
msg295094 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2017-06-03 18:24
One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, otherwise coverage shifts and we can't rely on any coverage metrics per-PR to know if code is increasing or decreasing coverage. Maybe having a ``-u deterministic`` resource to take out the randomness for coverage runs but leave the randomness for buildbots? This might also require tweaking our Travis tests as they currently use the buildbot make rule (I think).

P.S. I bet there are also some multiprocessing tests that go a bit overboard that we could consider scaling back, e.g. the coverage run skips a bunch of tests because they seem to process the entire stdlib.
msg295219 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-05 23:42
New changeset c53b13b270767948fddb58b287149c499f9a03c4 by Zachary Ware in branch 'master':
bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
https://github.com/python/cpython/commit/c53b13b270767948fddb58b287149c499f9a03c4
msg295224 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-06 00:10
New changeset d125738e93a35125d0edbe8070dd47160bcda829 by Zachary Ware in branch '3.6':
[3.6] bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
https://github.com/python/cpython/commit/d125738e93a35125d0edbe8070dd47160bcda829
msg295225 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-06 01:02
New changeset 58e5dd1a03b3ce421f07085f8cd6e8566a1ed197 by Zachary Ware in branch '3.5':
[3.5] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
https://github.com/python/cpython/commit/58e5dd1a03b3ce421f07085f8cd6e8566a1ed197
msg295226 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-06 01:03
New changeset ddf8f141f0181f9cac05c87383ea60887acffaec by Zachary Ware in branch '2.7':
[2.7] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
https://github.com/python/cpython/commit/ddf8f141f0181f9cac05c87383ea60887acffaec
msg295594 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 01:20
New changeset 42e3acda86829def9adc354fbee77597b849bf9e by Zachary Ware in branch 'master':
bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/42e3acda86829def9adc354fbee77597b849bf9e
msg295599 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 03:26
New changeset e380c19b7e88d97d3497016f1065d6922415db1c by Zachary Ware in branch '3.6':
[3.6] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/e380c19b7e88d97d3497016f1065d6922415db1c
msg295600 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 03:26
New changeset 91581d4225e91e581f44d60d0b87c80778c36d1d by Zachary Ware in branch '3.5':
[3.5] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/91581d4225e91e581f44d60d0b87c80778c36d1d
msg295601 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-06-10 03:27
New changeset 3c7b67bb34bc537b6c0dd9c6e1e4a07dc227b467 by Zachary Ware in branch '2.7':
[2.7] bpo-30417: Disable `cpu` resource on AppVeyor (GH-1951)
https://github.com/python/cpython/commit/3c7b67bb34bc537b6c0dd9c6e1e4a07dc227b467
msg295625 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-10 09:06
Thanks for taking care of our CI, Zach ;-) Buildbots, Travis CI job and AppVeyor are now fixed if I understood correctly, so I close the issue.


Serhiy Storchaka: "And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu"."

Hum, I have no real opinion on this issue. I think that the first main issue was test_tools, and disabling cpu on our CI has fixed the issue. I'm not sure that we should go deeper. *If* you consider thta that test_datetime is too slow, yeah, maybe add also a requirement on the "cpu" resource. But I don't recall to have seen "test_datetime" in the final "top 10 slowest tests" (even if I didn't look recently).


Brett Cannon: "One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, (...)"

Using a fixed list of filenames would benefit to everyone, not only to the CI. But someone has to select these files :-)

Brett: "Maybe having a ``-u deterministic`` resource to take out the randomness for coverage runs but leave the randomness for buildbots?"

We may start by using a fixed random seed for our coverage tests? And make sure that each test file starts with the same seed? IMHO coverage is a different issue, so I suggest to open a new issue if you consider that it's worth it to enhance the coverage CI job ;-)
History
Date User Action Args
2017-06-10 09:06:56hayposetstatus: open -> closed
resolution: fixed
messages: + msg295625

stage: resolved
2017-06-10 03:27:11zach.waresetmessages: + msg295601
2017-06-10 03:26:50zach.waresetmessages: + msg295600
2017-06-10 03:26:34zach.waresetmessages: + msg295599
2017-06-10 01:30:40zach.waresetpull_requests: + pull_request2120
2017-06-10 01:25:56zach.waresetpull_requests: + pull_request2119
2017-06-10 01:24:25zach.waresetpull_requests: + pull_request2118
2017-06-10 01:20:26zach.waresetmessages: + msg295594
2017-06-06 01:03:07zach.waresetmessages: + msg295226
2017-06-06 01:02:20zach.waresetmessages: + msg295225
2017-06-06 00:10:24zach.waresetmessages: + msg295224
2017-06-06 00:01:19zach.waresetpull_requests: + pull_request2034
2017-06-05 23:51:39zach.waresetpull_requests: + pull_request2033
2017-06-05 23:50:07zach.waresetpull_requests: + pull_request2032
2017-06-05 23:42:19zach.waresetmessages: + msg295219
2017-06-04 22:33:12zach.waresetpull_requests: + pull_request2023
2017-06-03 18:24:20brett.cannonsetnosy: + brett.cannon
messages: + msg295094
2017-06-03 17:17:26serhiy.storchakasetmessages: + msg295089
2017-06-03 16:41:56zach.waresetpull_requests: + pull_request2008
2017-06-03 16:37:37serhiy.storchakasetnosy: + serhiy.storchaka, pitrou, belopolsky
messages: + msg295086
2017-05-20 22:16:18zach.waresetnosy: + zach.ware
2017-05-20 22:06:09haypocreate