classification
Title: Failures/crashes in test_signal on some buildbots
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: 30807 Superseder:
Assigned To: Nosy List: db3l, haypo, koobs, pitrou
Priority: normal Keywords:

Created on 2017-06-28 22:23 by pitrou, last changed 2017-07-01 17:12 by pitrou. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 2488 merged pitrou, 2017-06-29 12:07
Messages (16)
msg297237 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 22:23
The test added for issue #30703 crashes on some Unix buildbots (FreeBSD 9.x, OS X Tiger, AIX).

Example here:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/275/steps/test/logs/stdio

As noted in the PR for that issue, some systems may have low resolution for setitimer(). For example, the FreeBSD man page says:

> Time values smaller than the resolution of the system clock are rounded up to this resolution (typically 10 milliseconds).
msg297239 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-28 22:26
koobs, perhaps you would like to give me temporary shell access to that machine?
msg297260 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-29 07:26
The process is even killed.

http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.x%203.x/builds/275/steps/test/logs/stdio

Re-running test 'test_signal' in verbose mode
test_enums (test.test_signal.GenericTests) ... ok
test_itimer_exc (test.test_signal.ItimerTest) ... ok
test_itimer_prof (test.test_signal.ItimerTest) ... ok
test_itimer_real (test.test_signal.ItimerTest) ... ok
test_itimer_virtual (test.test_signal.ItimerTest) ... ok
test_pthread_kill (test.test_signal.PendingSignalsTests) ... ok
test_pthread_kill_main_thread (test.test_signal.PendingSignalsTests) ... ok
test_pthread_sigmask (test.test_signal.PendingSignalsTests) ... ok
test_pthread_sigmask_arguments (test.test_signal.PendingSignalsTests) ... ok
test_sigpending (test.test_signal.PendingSignalsTests) ... ok
test_sigpending_empty (test.test_signal.PendingSignalsTests) ... ok
test_sigtimedwait (test.test_signal.PendingSignalsTests) ... ok
test_sigtimedwait_negative_timeout (test.test_signal.PendingSignalsTests) ... ok
test_sigtimedwait_poll (test.test_signal.PendingSignalsTests) ... ok
test_sigtimedwait_timeout (test.test_signal.PendingSignalsTests) ... ok
test_sigwait (test.test_signal.PendingSignalsTests) ... ok
test_sigwait_thread (test.test_signal.PendingSignalsTests) ... ok
test_sigwaitinfo (test.test_signal.PendingSignalsTests) ... ok
test_getsignal (test.test_signal.PosixTests) ... ok
test_interprocess_signal (test.test_signal.PosixTests) ... ok
test_out_of_range_signal_number_raises_error (test.test_signal.PosixTests) ... ok
test_setting_signal_handler_to_none_raises_error (test.test_signal.PosixTests) ... ok
test_siginterrupt_off (test.test_signal.SiginterruptTest) ... ok
test_siginterrupt_on (test.test_signal.SiginterruptTest) ... ok
test_without_siginterrupt (test.test_signal.SiginterruptTest) ... ok
test_stress_delivery_dependent (test.test_signal.StressTest) ... FAIL
test_stress_delivery_simultaneous (test.test_signal.StressTest) ... FAIL
test_invalid_fd (test.test_signal.WakeupFDTests) ... ok
test_invalid_socket (test.test_signal.WakeupFDTests) ... ok
test_set_wakeup_fd_blocking (test.test_signal.WakeupFDTests) ... ok
test_set_wakeup_fd_result (test.test_signal.WakeupFDTests) ... ok
test_set_wakeup_fd_socket_result (test.test_signal.WakeupFDTests) ... ok
test_pending (test.test_signal.WakeupSignalTests) ... *** [buildbottest] Signal 14
msg297267 - (view) Author: Kubilay Kocak (koobs) Date: 2017-06-29 11:13
@Antoine Happy to any time. PM me public key on IRC (freenode:koobs) and I'll reply with user@host:port
msg297271 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 14:40
New changeset f7d090c165f6cd3d008fe60c78e5324caef53f80 by Antoine Pitrou in branch 'master':
bpo-30796: Fix failures in signal delivery stress test (#2488)
https://github.com/python/cpython/commit/f7d090c165f6cd3d008fe60c78e5324caef53f80
msg297275 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 15:14
The stable buildbots seem fine now (problem fixed on FreeBSD 9, thank you Koobs!).
msg297278 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-06-29 15:31
http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/979/steps/test/logs/stdio

test_stress_delivery_dependent (test.test_signal.StressTest) ... Timeout (0:15:00)!
Thread 0x405acdc0 (most recent call first):
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 973 in measure_itimer_resolution
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 985 in decide_itimer_count
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/test_signal.py", line 1001 in test_stress_delivery_dependent
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 615 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 663 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/runner.py", line 176 in run
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 168 in test_runner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 169 in runtest_inner
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/runtest.py", line 137 in runtest
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 540 in _main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 85 in _run_code
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=2727.765081
msg297279 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:09
Hmm, that function only sends 20 signals in a row.  How can it timeout after 15 seconds?
msg297280 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:14
Hmm, it's not 15 seconds, it's 15 minutes...
msg297281 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:18
Ok, there's definitely something fishy here.  Look at the elapsed times:

0:16:33 load avg: 0.93 [281/406] test_ossaudiodev skipped -- running: test_signal (874 sec)
test_ossaudiodev skipped -- [Errno 2] No such file or directory: '/dev/dsp'
0:16:59 load avg: 0.95 [282/406/1] test_signal crashed (Exit code 1)
Timeout (0:15:00)!


faulthandler claims that 15 minutes have elapsed, but only 59-33 = 26 *seconds* have!
msg297282 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:29
x86 Tiger is fine too.
msg297283 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 16:31
David, would it be possible to have shell access to that machine?
msg297294 - (view) Author: David Bolen (db3l) Date: 2017-06-29 18:54
Antoine, yes.  Send me your public key (db3l.net at gmail) and I'll set it up.
msg297302 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-29 19:56
Ok, the timeout on the "Ubuntu Shared" buildbot is due to https://bugs.python.org/issue30807
msg297359 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-06-30 08:32
It seems fixing https://bugs.python.org/issue30807 has fixed the remaining buildbot issues.  I will close once everything is settled.
msg297485 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-01 17:12
New changeset 3024c0529077f5cff0b32dc84b5923c8fba99a87 by Antoine Pitrou in branch '3.6':
[3.6] bpo-30703: Improve signal delivery (GH-2415) (#2527)
https://github.com/python/cpython/commit/3024c0529077f5cff0b32dc84b5923c8fba99a87
History
Date User Action Args
2017-07-01 17:12:08pitrousetmessages: + msg297485
2017-06-30 15:41:29pitrousetstatus: open -> closed
resolution: fixed
stage: commit review -> resolved
2017-06-30 08:32:07pitrousetmessages: + msg297359
2017-06-29 19:56:25pitrousetmessages: + msg297302
2017-06-29 19:49:43pitrousetstage: needs patch -> commit review
versions: - Python 2.7, Python 3.5, Python 3.6
2017-06-29 19:49:24pitrousetdependencies: + setitimer() can disable timer by mistake
versions: + Python 2.7, Python 3.5, Python 3.6
2017-06-29 18:54:25db3lsetmessages: + msg297294
2017-06-29 16:31:36pitrousetmessages: + msg297283
2017-06-29 16:29:08pitrousetmessages: + msg297282
2017-06-29 16:18:34pitrousetmessages: + msg297281
2017-06-29 16:14:24pitrousetmessages: + msg297280
2017-06-29 16:09:03pitrousetnosy: + db3l
messages: + msg297279
2017-06-29 15:31:54hayposetmessages: + msg297278
2017-06-29 15:14:26pitrousetmessages: + msg297275
2017-06-29 14:40:16pitrousetmessages: + msg297271
2017-06-29 12:07:43pitrousetpull_requests: + pull_request2547
2017-06-29 11:13:24koobssetmessages: + msg297267
2017-06-29 07:26:44hayposetmessages: + msg297260
2017-06-28 22:26:16pitrousetmessages: + msg297239
2017-06-28 22:25:02pitrousetnosy: + koobs
2017-06-28 22:23:54pitroucreate