➜

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x
Type: Stage: resolved
Components: Tests Versions: Python 3.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: pitrou Nosy List: pitrou, serhiy.storchaka, vstinner, zach.ware
Priority: normal Keywords:

Created on 2017-07-04 15:16 by vstinner, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Messages (15)
msg297674 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-04 15:16 
Antoine: I told you that such stess-test is going to require work to polish it. So here is another example of failure. Enjoy :-)

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/534/steps/test/logs/stdio


== CPython 3.6.2rc1+ (heads/3.6:580cd5c, Jul 5 2017, 00:47:42) [GCC 4.9.2]
== Linux-3.16.0-4-amd64-x86_64-with-debian-8.7 little-endian
...
== CPU count: 1
...
Run tests in parallel using 2 child processes
0:11:40 load avg: 1.29 [383/405/1] test_signal failed -- running: test_datetime (38 sec)
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4602 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1086, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8206 != 10000 : Some signals were lost
msg297732 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-07-05 12:38 
Another fail. The test pass when run again later.

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.6/builds/538/steps/test/logs/stdio

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.6.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1053, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 4073 != 10000 : Some signals were lost
msg299948 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-08 21:09 
Hey Antoine, as expected, your test fails randomly on the CI :-(

Another failure:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/829/steps/test/logs/stdio

test_stress_delivery_dependent (test.test_signal.StressTest) ... Timeout (0:15:00)!
Thread 0xb74bd700 (most recent call first):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 973 in measure_itimer_resolution
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 985 in decide_itimer_count
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_signal.py", line 1001 in test_stress_delivery_dependent
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 615 in run
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 663 in __call__
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 122 in run
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/suite.py", line 84 in __call__
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/runner.py", line 176 in run
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/__init__.py", line 1896 in _run_suite
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/support/__init__.py", line 1940 in run_unittest
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 168 in test_runner
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 169 in runtest_inner
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/runtest.py", line 137 in runtest
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 290 in rerun_failed_tests
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 540 in _main
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 510 in main
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/libregrtest/main.py", line 585 in main
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/__main__.py", line 2 in <module>
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 85 in _run_code
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/runpy.py", line 193 in _run_module_as_main
msg299985 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-09 10:03 
Hmm.  Perhaps someone can give me temporary shell access to one of those buildbots?
msg299986 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-09 10:31 
Antoine: "Hmm.  Perhaps someone can give me temporary shell access to one of those buildbots?"

Contact directly the owners:

* AMD64 Debian root 3.6: Chris Angelico <rosuav@gmail.com> 
* x86 Gentoo Non-Debug with X 3.x: Zachary Ware <zachary.ware+bb@gmail.com>
msg300017 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2017-08-09 15:04 
> Hmm.  Perhaps someone can give me temporary shell access to one of those buildbots?

Check your @python.org inbox.
msg300181 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-08-11 22:14 
Victor, Zachary, http://buildbot.python.org/all/builders/x86%20Gentoo%20Non-Debug%20with%20X%203.x/builds/830 corresponds to a very old changeset (more than one month old) before itimer() was fixed in 729780a810bbcb12b245a1b652302a601fc9f6fd.
msg300182 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-11 22:19 
Oh, Zach's Gentoo buildbot was offline for a long time. Maybe it ran tests
on old commits? What about the first messages of this issue on the other
buildbot? Maybe it's also outdated? I am fine with closing the issue if you
fixed a bug in the meanwhile. Don't worry, I will reopen it if it comes
back :-D
msg300550 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-18 23:23 
test_stress_delivery_dependent() failed on AMD64 Debian root 3.x too, it's not specific to 3.6:

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1192/steps/test/logs/stdio

http://buildbot.python.org/all/builders/AMD64%20Debian%20root%203.x/builds/1191/steps/test/logs/stdio
msg318313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-31 14:00 
Recent failure on AMD64 Debian root 3.7:

http://buildbot.python.org/all/#/builders/127/builds/361

0:03:33 load avg: 1.38 [140/415/1] test_signal failed

======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3586 != 10000 : Some signals were lost

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.7.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1149, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 8884 != 10000 : Some signals were lost
msg320891 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-02 15:19 
I can reproduce the failure on multiple OSes in VirtaulBox when run test_signal in parallel with test_regrtest:

./python -m test -j2 test_signal  test_regrtest test_regrtest test_regrtest
Run tests in parallel using 2 child processes
0:00:29 load avg: 4.24 [1/4] test_regrtest passed
0:00:58 load avg: 4.96 [2/4] test_regrtest passed -- running: test_signal (58 sec 728 ms)
0:01:00 load avg: 4.96 [3/4/1] test_signal failed
test test_signal failed -- Traceback (most recent call last):
  File "/home/serhiy/py/cpython3.7/Lib/test/test_signal.py", line 1116, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5073 != 10000 : Some signals were lost

But can't reproduce on hardware.
msg320900 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-02 16:38 
After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.
msg321019 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-04 08:17 
> After increasing the number of CPUs in the virtual machine the failure no longer reproduced. It is reproducible only with a single CPU.

The test just failed on AMD64 Debian root 3.x and according to test.pythoninfo, this machine has a single CPU:

os.cpu_count: 1

http://buildbot.python.org/all/#/builders/27/builds/1247

...
== CPU count: 1
== encodings: locale=UTF-8, FS=utf-8
Using random seed 6423496
Run tests in parallel using 2 child processes
0:00:00 load avg: 0.71 [  1/417] test_genericclass passed
0:00:00 load avg: 0.71 [  2/417] test_copyreg passed
0:00:03 load avg: 0.71 [  3/417] test_os passed
stty: standard input: Inappropriate ioctl for device
running: test_tokenize (32 sec 627 ms), test_signal (30 sec 11 ms)
0:00:51 load avg: 1.32 [  4/417] test_tokenize passed (50 sec 821 ms) -- running: test_signal (48 sec 506 ms)
0:00:54 load avg: 1.32 [  5/417/1] test_signal failed
...
test_stress_delivery_dependent (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000044 s.
FAIL
test_stress_delivery_simultaneous (test.test_signal.StressTest) ... detected median itimer() resolution: 0.000032 s.
ok
...
======================================================================
FAIL: test_stress_delivery_dependent (test.test_signal.StressTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/buildarea/3.x.angelico-debian-amd64/build/Lib/test/test_signal.py", line 1156, in test_stress_delivery_dependent
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 3617 != 10000 : Some signals were lost
msg394492 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-05-26 22:47 
I didn't see this failure recently, I close the issue.
msg401380 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-09-08 12:47 
We also got this error randomly on an internal s390x Red Hat build server:

======================================================================
FAIL: test_stress_delivery_simultaneous (test.test_signal.StressTest)
This test uses simultaneous signal handlers.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib64/python3.9/test/test_signal.py", line 1245, in test_stress_delivery_simultaneous
    self.assertEqual(len(sigs), N, "Some signals were lost")
AssertionError: 5936 != 10000 : Some signals were lost
History
Date User Action Args
2022-04-11 14:58:48adminsetgithub: 75032
2021-09-08 12:47:26vstinnersetmessages: + msg401380
2021-05-26 22:47:39vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg394492

stage: resolved
2018-07-04 08:17:20vstinnersetmessages: + msg321019
2018-07-02 16:38:25serhiy.storchakasetmessages: + msg320900
2018-07-02 15:19:43serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg320891
2018-05-31 14:00:52vstinnersetmessages: + msg318313
2017-08-18 23:23:33vstinnersetmessages: + msg300550
2017-08-18 23:21:59vstinnersettitle: test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6 -> test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6/3.x
2017-08-11 22:19:53vstinnersetmessages: + msg300182
2017-08-11 22:14:59pitrousetmessages: + msg300181
2017-08-09 15:04:36zach.waresetmessages: + msg300017
2017-08-09 10:31:37vstinnersetmessages: + msg299986
2017-08-09 10:03:42pitrousetnosy: + zach.ware
messages: + msg299985
2017-08-08 21:09:37vstinnersetmessages: + msg299948
2017-07-05 12:38:57vstinnersetmessages: + msg297732
title: test_stress_delivery_dependent() randomly fails on AMD64 Debian root 3.6 -> test_stress_delivery_dependent() of test_signal randomly fails on AMD64 Debian root 3.6
2017-07-04 15:16:51vstinnercreate