classification
Title: test_create_at_shutdown_with_encoding() of test_io hangs on "SPARC Solaris 10 OpenCSW 3.x"
Type: Stage:
Components: Tests Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, flub, jcea, larry, serhiy.storchaka, vstinner
Priority: Keywords: buildbot

Created on 2014-02-18 09:48 by vstinner, last changed 2014-09-26 02:43 by jcea. This issue is now closed.

Messages (10)
msg211502 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-18 09:48
http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/813/steps/test/logs/stdio

[162/389/1] test_io
Timeout (1:00:00)!
Current thread 0x00000001 (most recent call first):
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 1378 in _execute_child
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 850 in __init__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/script_helper.py", line 43 in _assert_python
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/script_helper.py", line 69 in assert_python_ok
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_io.py", line 2667 in _check_create_at_shutdown
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_io.py", line 2680 in test_create_at_shutdown_with_encoding
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/case.py", line 574 in run
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/case.py", line 622 in __call__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 125 in run
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 87 in __call__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 125 in run
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 87 in __call__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 125 in run
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/suite.py", line 87 in __call__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/unittest/runner.py", line 168 in run
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/support/__init__.py", line 1685 in _run_suite
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/support/__init__.py", line 1719 in run_unittest
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/regrtest.py", line 1277 in <lambda>
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/regrtest.py", line 1278 in runtest_inner
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/regrtest.py", line 967 in runtest
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/regrtest.py", line 763 in main
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/regrtest.py", line 1562 in main_in_temp_cwd
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/__main__.py", line 3 in <module>
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/runpy.py", line 86 in _run_code
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/runpy.py", line 171 in _run_module_as_main
msg211511 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-18 12:57
Build #810 and earlier did not hanged. Build #811 (http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/811) hanged on test_multiprocessing_spawn, build #812 (http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/812) hanged on test_tempfile, and build #813 (http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/813) hanged on test_io.

May be the culprit of this is one of changesets: b328f8ccbccf (issue16251), 52ab9e1ff46a (issue20621), or (unlikely) 180e4b678003 (issue20320).
msg211574 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-19 00:50
It looks like a serious regression.
msg211575 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-19 00:55
> It looks like a serious regression.

Oops, I am no more sure. Tests of the build 814 didn't hang.
msg211576 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-02-19 00:56
For what it's worth I don't mind the noise, if indeed it's a false alarm.  I'm glad you're keeping an eye on the buildbots too.
msg211577 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-19 00:58
Oh... A single build on this buildbot takes 10 hours. Maybe the buildbot is slow or busy? Maybe the timeout on this buildbot should be increased?


Other hangs:

http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/803/steps/test/logs/stdio

[ 93/389/2] test_multiprocessing_main_handling
Timeout (1:00:00)!
Current thread 0x00000001 (most recent call first):
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 1378 in _execute_child
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 850 in __init__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/script_helper.py", line 43 in _assert_python
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/script_helper.py", line 69 in assert_python_ok
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_multiprocessing_main_handling.py", line 152 in _check_script
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_multiprocessing_main_handling.py", line 221 in test_zipfile_compiled

http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/801/steps/test/logs/stdio

[320/389/3] test_tools
Timeout (1:00:00)!
Current thread 0x00000001 (most recent call first):
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 1378 in _execute_child
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/subprocess.py", line 850 in __init__
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_tools.py", line 54 in pindent
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_tools.py", line 106 in pindent_test
  File "/export/home/buildbot/buildarea/3.x.bruynooghe-solaris-csw/build/Lib/test/test_tools.py", line 352 in test_empty_line
msg211829 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-21 08:54
I sent an email to Floris Bruynooghe, owner of the buildbot:

Hi,

I noticed "Timeout (1:00:00)!" errors on your buildbot "SPARC Solaris
10 OpenCSW 3.x". It looks like the buildbot takes 10 hour for one
build, whereas my PC takes less than 10 minutes (especially when tests
are run in parallel). I don't really care of performances, but more on
the result.

Could you please modify the timeout, for example to set it to 2 hours?

The "test" step of buildbot uses this command:

 argv: ['make', 'buildbottest', 'TESTOPTS=', 'TESTPYTHONOPTS=',
'TESTTIMEOUT=3600']

Replace 3600 with 7200 (seconds = 2 hours).

Related Python issue:
http://bugs.python.org/issue20671

Thanks,
Victor Stinner aka haypo
msg211869 - (view) Author: Floris Bruynooghe (flub) Date: 2014-02-21 15:55
Turns out that the timeout is configured in the buildmaster's master.cfg which Antoine Pitrou has kindly done.  It should also run tests a bit more parallel now which will hopefully reduce the 10h runtime a bit, but it remains a slow box.
msg211899 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-02-22 00:29
The buildbot now runs tests with -j16 (hum, I fact there are -j1 and -j16 in the command line), it looks faster: between 3 and 4 hours per build.

The current build is running since less than 1 hour, but the test output already contains "Timeout (9:55:00)!". Strange that it displays 9h55 whereas the whole build is running since less than 1 hour? By the way, why so long timeout? I proposed 2 hours, not 10 hours :-)

http://buildbot.python.org/all/builders/SPARC%20Solaris%2010%20OpenCSW%203.x/builds/826/steps/test/logs/stdio
msg213921 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-17 23:47
The configuration of the buildbot has been changed. Tests are now running with -j16 (16 processes) and the test suite takes 1 hour 30 instead of 10 hours.

This issue has been fixed with the new configuration.
History
Date User Action Args
2014-09-26 02:43:40jceasetnosy: + jcea
2014-03-17 23:47:03vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg213921
2014-02-22 00:29:10vstinnersetmessages: + msg211899
2014-02-21 15:55:32flubsetnosy: + flub
messages: + msg211869
2014-02-21 08:54:38vstinnersetmessages: + msg211829
2014-02-19 00:58:25vstinnersetmessages: + msg211577
2014-02-19 00:56:27larrysetmessages: + msg211576
2014-02-19 00:55:09vstinnersetpriority: release blocker ->

messages: + msg211575
2014-02-19 00:52:32vstinnersetnosy: + benjamin.peterson
2014-02-19 00:50:08vstinnersetpriority: normal -> release blocker
nosy: + larry
messages: + msg211574

2014-02-18 12:57:04serhiy.storchakasetmessages: + msg211511
2014-02-18 09:48:46vstinnercreate