Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[buildbot] Disable the cpu resources on slow buildbots #74602

Closed
vstinner opened this issue May 20, 2017 · 13 comments
Closed

[buildbot] Disable the cpu resources on slow buildbots #74602

vstinner opened this issue May 20, 2017 · 13 comments
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 30417
Nosy @brettcannon, @abalkin, @pitrou, @vstinner, @zware, @serhiy-storchaka
PRs
  • bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis #1928
  • bpo-30417: Disable cpu resource on AppVeyor #1951
  • [3.6] bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928) #1961
  • [3.5] bpo-30417: Disable 'cpu' resource on Travis (GH-1928) #1962
  • [2.7] bpo-30417: Disable 'cpu' resource on Travis (GH-1928) #1964
  • [3.6] bpo-30417: Disable cpu resource on AppVeyor (GH-1951) #2057
  • [3.5] bpo-30417: Disable cpu resource on AppVeyor (GH-1951). #2058
  • [2.7] bpo-30417: Disable cpu resource on AppVeyor (GH-1951) #2059
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-06-10.09:06:56.020>
created_at = <Date 2017-05-20.22:06:09.443>
labels = ['3.7', 'tests']
title = '[buildbot] Disable the cpu resources on slow buildbots'
updated_at = <Date 2017-06-10.09:06:56.019>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-06-10.09:06:56.019>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-06-10.09:06:56.020>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-05-20.22:06:09.443>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 30417
keywords = []
message_count = 13.0
messages = ['294052', '295086', '295089', '295094', '295219', '295224', '295225', '295226', '295594', '295599', '295600', '295601', '295625']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'belopolsky', 'pitrou', 'vstinner', 'zach.ware', 'serhiy.storchaka']
pr_nums = ['1928', '1951', '1961', '1962', '1964', '2057', '2058', '2059']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue30417'
versions = ['Python 3.7']
    @vstinner
    Copy link
    Member Author

    Follow-up of the issue bpo-30314. Copy of Serhiy Storchaka's msg294048:

    test_tools is so slow because it proceeds *all* Python files. This controlled by the "cpu" resource (only 10 random files are chosen if it is disabled).

    Maybe disable the "cpu" resource on slow buildbots?

    Good point, I like the idea.

    It seems like regrtest accepts -u all,-cpu option.

    @vstinner vstinner added 3.7 (EOL) end of life tests Tests in the Lib/test dir labels May 20, 2017
    @serhiy-storchaka
    Copy link
    Member

    And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu".

    @serhiy-storchaka
    Copy link
    Member

    This made tests faster by a third!

    @brettcannon
    Copy link
    Member

    One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, otherwise coverage shifts and we can't rely on any coverage metrics per-PR to know if code is increasing or decreasing coverage. Maybe having a -u deterministic resource to take out the randomness for coverage runs but leave the randomness for buildbots? This might also require tweaking our Travis tests as they currently use the buildbot make rule (I think).

    P.S. I bet there are also some multiprocessing tests that go a bit overboard that we could consider scaling back, e.g. the coverage run skips a bunch of tests because they seem to process the entire stdlib.

    @zware
    Copy link
    Member

    zware commented Jun 5, 2017

    New changeset c53b13b by Zachary Ware in branch 'master':
    bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
    c53b13b

    @zware
    Copy link
    Member

    zware commented Jun 6, 2017

    New changeset d125738 by Zachary Ware in branch '3.6':
    [3.6] bpo-30417: Disable 'cpu' and 'tzdata' resources on Travis (GH-1928)
    d125738

    @zware
    Copy link
    Member

    zware commented Jun 6, 2017

    New changeset 58e5dd1 by Zachary Ware in branch '3.5':
    [3.5] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
    58e5dd1

    @zware
    Copy link
    Member

    zware commented Jun 6, 2017

    New changeset ddf8f14 by Zachary Ware in branch '2.7':
    [2.7] bpo-30417: Disable 'cpu' resource on Travis (GH-1928)
    ddf8f14

    @zware
    Copy link
    Member

    zware commented Jun 10, 2017

    New changeset 42e3acd by Zachary Ware in branch 'master':
    bpo-30417: Disable cpu resource on AppVeyor (GH-1951)
    42e3acd

    @zware
    Copy link
    Member

    zware commented Jun 10, 2017

    New changeset e380c19 by Zachary Ware in branch '3.6':
    [3.6] bpo-30417: Disable cpu resource on AppVeyor (GH-1951)
    e380c19

    @zware
    Copy link
    Member

    zware commented Jun 10, 2017

    New changeset 91581d4 by Zachary Ware in branch '3.5':
    [3.5] bpo-30417: Disable cpu resource on AppVeyor (GH-1951)
    91581d4

    @zware
    Copy link
    Member

    zware commented Jun 10, 2017

    New changeset 3c7b67b by Zachary Ware in branch '2.7':
    [2.7] bpo-30417: Disable cpu resource on AppVeyor (GH-1951)
    3c7b67b

    @vstinner
    Copy link
    Member Author

    Thanks for taking care of our CI, Zach ;-) Buildbots, Travis CI job and AppVeyor are now fixed if I understood correctly, so I close the issue.

    Serhiy Storchaka: "And disable the "tzdata" resource for skipping generated tests for all timezones in test_datetime. Or make them requiring not only "tzdata", but "cpu"."

    Hum, I have no real opinion on this issue. I think that the first main issue was test_tools, and disabling cpu on our CI has fixed the issue. I'm not sure that we should go deeper. *If* you consider thta that test_datetime is too slow, yeah, maybe add also a requirement on the "cpu" resource. But I don't recall to have seen "test_datetime" in the final "top 10 slowest tests" (even if I didn't look recently).

    Brett Cannon: "One thing to note is if we want to speed up things like the coverage run on Travis we may want to make this optionally more deterministic rather than fully random for the 10 selected files, (...)"

    Using a fixed list of filenames would benefit to everyone, not only to the CI. But someone has to select these files :-)

    Brett: "Maybe having a -u deterministic resource to take out the randomness for coverage runs but leave the randomness for buildbots?"

    We may start by using a fixed random seed for our coverage tests? And make sure that each test file starts with the same seed? IMHO coverage is a different issue, so I suggest to open a new issue if you consider that it's worth it to enhance the coverage CI job ;-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    4 participants
    @brettcannon @vstinner @serhiy-storchaka @zware