classification
Title: test_largefile.TestSocketSendfile.test_it() failed with 15 min timeout on AMD64 FreeBSD 10-STABLE Non-Debug 3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, pablogsal, vstinner
Priority: normal Keywords:

Created on 2019-10-08 21:14 by vstinner, last changed 2019-10-10 09:21 by vstinner. This issue is now closed.

Messages (6)
msg354234 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 21:14
It seems like the 15 min timeout for the overall test_largefile is too short for the very slow AMD64 FreeBSD 10-STABLE Non-Debug 3.x buildbot.

The test creates two files of 2.5 GB on the disk.

One solution is to skip the test on this buildbot worker. For example, using a test resource like "largefile","cpu" resource, or another one. Maybe only skip TestSocketSendfile.test_it(), or even skip for the whole file. Currently, the whole file requires the "largefile" resource, but only on Windows and macOS.

Right now, the buildbot worker runs the test suite using:

   ./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 -j4 

So all resources are enabled.

--

https://buildbot.python.org/all/#/builders/167/builds/1695

test_it (test.test_largefile.TestCopyfile) ... ok
Timeout (0:15:00)!
Thread 0x0000000806c95400 (most recent call first):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/test_largefile.py", line 193 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/threading.py", line 882 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/threading.py", line 944 in _bootstrap_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/threading.py", line 902 in _bootstrap

Thread 0x0000000801c06400 (most recent call first):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/socket.py", line 386 in _sendfile_use_sendfile
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/socket.py", line 483 in sendfile
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/test_largefile.py", line 207 in test_it
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/case.py", line 616 in _callTestMethod
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/case.py", line 659 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/case.py", line 719 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/unittest/runner.py", line 176 in run
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/support/__init__.py", line 2032 in _run_suite
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/support/__init__.py", line 2128 in run_unittest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/runtest.py", line 209 in _test_module
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/runtest.py", line 234 in _runtest_inner2
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/runtest.py", line 270 in _runtest_inner
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/runtest.py", line 153 in _runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/runtest.py", line 193 in runtest
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/main.py", line 318 in rerun_failed_tests
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/main.py", line 691 in _main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/main.py", line 634 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/libregrtest/main.py", line 712 in main
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/test/__main__.py", line 2 in <module>
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/runpy.py", line 85 in _run_code
  File "/usr/home/buildbot/python/3.x.koobs-freebsd10.nondebug/build/Lib/runpy.py", line 192 in _run_module_as_main
test_it (test.test_largefile.TestSocketSendfile) ... *** Error code 1
msg354235 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 21:15
On the previous build, the test failed with timeout while other tests were running in parallel, and then the test passed ... in 14 min 43 seconds... which is very close to the limit of 15 minutes :-)
msg354236 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-08 21:16
Another approach is to change the timeout of 15 min for this super slow buildbot. For example, set it to 20 min or 30 min.
msg354327 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-10-10 07:53
> Another approach is to change the timeout of 15 min for this super slow buildbot. For example, set it to 20 min or 30 min.

I think this makes more sense (set timeout to 25 min).
msg354332 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-10 08:10
I changed AMD64 FreeBSD 10-STABLE timeout from 15 to 30 minutes.

https://github.com/python/buildmaster-config/commit/c3fe0b1b65d94c115bbbb0521dda4daba02775dc
msg354350 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-10-10 09:19
Thanks.
History
Date User Action Args
2019-10-10 09:21:05vstinnersetstatus: open -> closed
resolution: fixed
stage: resolved
2019-10-10 09:19:52giampaolo.rodolasetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg354350

stage: resolved -> (no value)
2019-10-10 08:10:52vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg354332

stage: resolved
2019-10-10 07:53:19giampaolo.rodolasetmessages: + msg354327
2019-10-08 21:16:19vstinnersetmessages: + msg354236
2019-10-08 21:15:28vstinnersetmessages: + msg354235
2019-10-08 21:14:26vstinnercreate