classification
Title: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 buildbot
Type: Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: serhiy.storchaka, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-05-09 14:56 by vstinner, last changed 2017-06-28 00:51 by vstinner. This issue is now closed.

Messages (5)
msg293316 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-09 14:56
We should give more than 15 min to the "x86 Tiger 3.6" buildbot, since test_tools was killed by the timeout. On the previous build, test_tools took min 25 sec.

The global 15 min timeout was designed to detect deadlocks issues, but here it's just that the buildbot is sometimes too slow.

Maybe we should increase the timeout to 30 min on all buildbots, but also increase the timeout of the "tests" step to long than 1 hour? Otherwise, the test can be killed by buildbot without logging the Python traceback.

Maybe we can configured buildbot to first send a signal (SIGUSR1? SIGINT?) before killing the process to dump a traceback, but that's something different.

---
0:15:54 [ 52/405] test_tools crashed -- running: test_zipfile (178 sec)
Timeout (0:15:00)!
Thread 0xa000d000 (most recent call first):
  File "/Users/db3l/buildarea/3.6.bolen-tiger/build/Lib/ast.py", line 101 in <genexpr>
  ...
  File "/Users/db3l/buildarea/3.6.bolen-tiger/build/Lib/test/test_tools/test_unparse.py", line 124 in assertASTEqual
  ...
---
http://buildbot.python.org/all/builders/x86%20Tiger%203.6/builds/349/steps/test/logs/stdio

See also bpo-30172: "test_tools takes longer than 5 minutes on some buildbots".
msg294047 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-20 18:43
PR: https://github.com/python/buildmaster-config/pull/8
msg294048 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-05-20 18:52
test_tools is so slow because it proceeds *all* Python files. This controlled by the "cpu" resource (only 10 random files are chosen if it is disabled).

Maybe disable the "cpu" resource on slow buildbots?
msg294053 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-05-20 22:07
I merged my PR and applied the new buildbot configuration. I will try to check if the config was applied correctly.

Serhiy: "test_tools is so slow because it proceeds *all* Python files. This controlled by the "cpu" resource (only 10 random files are chosen if it is disabled)."

I created the bpo-30417 to track this idea.
msg297088 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-06-28 00:51
The timeout of buildbot has been adjusted (my PR https://github.com/python/buildmaster-config/pull/8 has been merged).

The cpu resource has been disabled on Travis CI to make it faster.

This issue can now been fixed.
History
Date User Action Args
2017-06-28 00:51:27vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg297088

stage: resolved
2017-05-20 22:14:11zach.waresetnosy: + zach.ware
2017-05-20 22:07:22vstinnersetmessages: + msg294053
components: + Tests
versions: + Python 3.7
2017-05-20 18:52:29serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg294048
2017-05-20 18:43:04vstinnersetmessages: + msg294047
2017-05-09 14:56:48vstinnercreate