classification
Title: test_pydoc hangs sometimes on 3.6 and master branches on ARMv7 Ubuntu 3.6/3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gregory.p.smith, haypo
Priority: normal Keywords: buildbot

Created on 2017-06-26 09:39 by haypo, last changed 2017-09-14 21:29 by haypo.

Messages (8)
msg296861 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-26 09:39
> ./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --timeout=900 -j2 -j8

faulthandler was supposed to kill test_pydoc after 15 minutes, and regrtest is supposed to log "running: pydoc (since xxx seconds)" every 30 seconds. Something gone bad. Buildbot killed regrtest because it didn't produce output during 20 minutes.

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/968/steps/test/logs/stdio

running: test_pydoc (420 sec), test_datetime (234 sec), test_tools (258 sec)
0:11:42 load avg: 3.90 [404/406] test_datetime passed (240 sec) -- running: test_pydoc (428 sec), test_tools (266 sec)
running: test_pydoc (458 sec), test_tools (296 sec)
0:12:13 load avg: 3.15 [405/406] test_tools passed (296 sec) -- running: test_pydoc (459 sec)

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j8', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1936.512343
msg296953 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-26 21:54
Similar issue:

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/800/steps/test/logs/stdio

...
1:49:04 [403/406/1] test_compile passed -- running: test_subprocess (4969 sec)
1:49:10 [404/406/1] test_strptime passed -- running: test_subprocess (4975 sec)
1:49:14 [405/406/1] test_unicode_file passed -- running: test_subprocess (4980 sec)

command timed out: 1200 seconds without output running ['Tools\\buildbot\\test.bat', '-j2', '--timeout', '900'], attempting to kill
program finished with exit code 1
elapsedTime=7768.278000
msg297600 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-03 15:02
Maybe the commit cdee3f14f7f4c995e7eedb0bf6a67e260c739f7d fixed test_subprocess on the x86 Windows7 3.x buildbot.
msg297601 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-03 15:03
I have no idea for the test_pydoc bug, but it was only seen once. I close the issue, but I may reopen it if I see again these bugs.
msg298320 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-13 23:14
Another fail, so I reopen the issue:

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.6/builds/563

...
0:11:30 load avg: 4.45 [403/405] test_lib2to3 passed (248 sec) -- running: test_tools (274 sec), test_pydoc (411 sec)
running: test_tools (304 sec), test_pydoc (441 sec)
running: test_tools (334 sec), test_pydoc (471 sec)
0:12:34 load avg: 2.83 [404/405] test_tools passed (338 sec) -- running: test_pydoc (476 sec)

command timed out: 1200 seconds without output ...
msg298948 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-07-24 09:51
It seems like test_pydoc only hangs on ARMv7 Ubuntu 3.x. Why faulthandler doesn't dump a traceback? :-(

faulthandler is supposed to dump a traceback after 900 seconds. Here, regrtest was killed after 1200 seconds and test_pydoc was already running for 423 seconds.

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/1094/steps/test/logs/stdio

./python  ./Tools/scripts/run_tests.py -j 1 -u all -W --slowest --fail-env-changed --timeout=900 -j2 -j8
...
0:10:28 load avg: 6.21 [403/406] test_venv passed (34 sec) -- running: test_pydoc (400 sec), test_pickle (47 sec), test_tools (338 sec)
0:10:32 load avg: 5.95 [404/406] test_pickle passed (51 sec) -- running: test_pydoc (405 sec), test_tools (343 sec)
0:10:51 load avg: 4.83 [405/406] test_tools passed (361 sec) -- running: test_pydoc (423 sec)

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j8', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
msg301875 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-11 13:31
http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.x/builds/1337/steps/test/logs/stdio

The issue still exists:

0:10:38 load avg: 7.31 [403/405] test_multiprocessing_forkserver passed (110 sec) -- running: test_pydoc (590 sec), test_buffer (55 sec)
0:10:41 load avg: 6.88 [404/405] test_buffer passed (57 sec) -- running: test_pydoc (593 sec)

command timed out: 1200 seconds without output running ['make', 'buildbottest', 'TESTOPTS=-j2 -j8', 'TESTPYTHONOPTS=', 'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1843.787084
msg302212 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-09-14 21:29
Python 3.6 is impacted as well on the same buildbot slave, ARMv7 Ubuntu 3.6:

http://buildbot.python.org/all/builders/ARMv7%20Ubuntu%203.6/builds/736/steps/test/logs/stdio
History
Date User Action Args
2017-09-14 21:29:04hayposetmessages: + msg302212
title: test_pydoc hangs sometimes on 3.6 and master branches on ARMv7 Ubuntu 3.x -> test_pydoc hangs sometimes on 3.6 and master branches on ARMv7 Ubuntu 3.6/3.x
2017-09-11 13:31:31hayposetmessages: + msg301875
2017-07-24 09:51:41hayposetmessages: + msg298948
2017-07-24 09:50:06hayposettitle: test_pydoc hangs sometimes on 3.6 and master branches -> test_pydoc hangs sometimes on 3.6 and master branches on ARMv7 Ubuntu 3.x
2017-07-13 23:14:04hayposetstatus: closed -> open
versions: + Python 3.6
title: regrtest hangs sometimes on the master branch (test_pydoc? test_subprocess?) -> test_pydoc hangs sometimes on 3.6 and master branches
messages: + msg298320

resolution: fixed ->
2017-07-03 15:03:20hayposetstatus: open -> closed
resolution: fixed
messages: + msg297601

stage: resolved
2017-07-03 15:02:18hayposetmessages: + msg297600
2017-06-27 21:43:25hayposettitle: regrtest hangs sometimes on the master branch (test_pydoc? test_subprocess?à) -> regrtest hangs sometimes on the master branch (test_pydoc? test_subprocess?)
2017-06-27 21:43:22hayposettitle: test_pydoc (regrtest?) hangs on some buildbots -> regrtest hangs sometimes on the master branch (test_pydoc? test_subprocess?à)
2017-06-27 00:09:47gregory.p.smithsettitle: test_pydoc (regrtest?) hangs on ARMv7 Ubuntu 3.x -> test_pydoc (regrtest?) hangs on some buildbots
2017-06-26 21:54:04hayposetmessages: + msg296953
2017-06-26 09:41:09hayposetnosy: + gregory.p.smith
2017-06-26 09:39:54haypocreate