classification
Title: test_compile hangs in AMD Ubuntu buildbots
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: pablogsal
Priority: normal Keywords:

Created on 2018-07-31 12:48 by pablogsal, last changed 2021-07-13 19:52 by pablogsal. This issue is now closed.

Messages (3)
msg322766 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-07-31 12:48
Test compile is hanging or failing in AMD(64) Ubuntu buildbots. For example:

https://buildbot.python.org/all/#/builders/157/builds/77
https://buildbot.python.org/all/#/builders/154/builds/90

Example errors:

running: test_tools (3 min 49 sec), test_compile (3 min 47 sec)
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/__main__.py", line 2, in <module>
    main()
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 584, in main
    Regrtest().main(tests=tests, **kwargs)
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 531, in main
    self._main(tests, kwargs)
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 566, in _main
    self.run_tests()
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 486, in run_tests
    run_tests_multiprocess(self)
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 181, in run_tests_multiprocess
    faulthandler.dump_traceback_later(test_timeout, exit=True)
RuntimeError: unable to start watchdog thread
command timed out: 1200 seconds without output running [b'make', b'buildbottest', b'TESTOPTS=-j2', b'TESTPYTHONOPTS=', b'TESTTIMEOUT=900'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=2780.229688

and

running: test_multiprocessing_forkserver (3 min 49 sec), test_compile (5 min 16 sec)
Makefile:1086: recipe for target 'buildbottest' failed
Timeout (0:15:00)!
Thread 0x00007ff09bab5700 (most recent call first):
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/selectors.py", line 415 in select
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/subprocess.py", line 1675 in _communicate
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/subprocess.py", line 933 in communicate
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 56 in run_test_in_subprocess
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 120 in _runtest
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 144 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/threading.py", line 917 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/threading.py", line 885 in _bootstrap
Thread 0x00007ff09c2b6700 (most recent call first):
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/selectors.py", line 415 in select
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/subprocess.py", line 1675 in _communicate
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/subprocess.py", line 933 in communicate
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 56 in run_test_in_subprocess
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 120 in _runtest
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 144 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/threading.py", line 917 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/threading.py", line 885 in _bootstrap
Thread 0x00007ff0a2ac7080 (most recent call first):
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest_mp.py", line 181 in run_tests_multiprocess
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 486 in run_tests
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 566 in _main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 531 in main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 584 in main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/runpy.py", line 85 in _run_code
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
program finished with exit code 2
elapsedTime=3446.164951

In subsequent runs of the buildbots test_compile emits a warning because it has ben re-ran:

https://buildbot.python.org/all/#/builders/154/builds/91
https://buildbot.python.org/all/#/builders/157/builds/80

The issue could be transient, but it seems that this could point at some other issue.
msg322935 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-08-02 10:03
New failure on AMD64 Ubuntu 3.x:

https://buildbot.python.org/all/#/builders/154/builds/100

test_single_statement (test.test_compile.TestSpecifics) ... ok
Timeout (0:15:00)!
Thread 0x00007f30435d3080 (most recent call first):
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/test_compile.py", line 671 in test_stack_overflow
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/case.py", line 615 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/case.py", line 663 in __call__
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/unittest/runner.py", line 176 in run
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/support/__init__.py", line 1883 in _run_suite
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/support/__init__.py", line 1973 in run_unittest
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 286 in rerun_failed_tests
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 570 in _main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 531 in main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 584 in main
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/runpy.py", line 85 in _run_code
  File "/home/buildbot/buildarea/3.x.einat-ubuntu/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
msg322939 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2018-08-02 10:14
New failure on AMD64 Ubuntu 3.7:

test_single_statement (test.test_compile.TestSpecifics) ... ok
Timeout (0:15:00)!
Thread 0x00007f5f41d9e080 (most recent call first):
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/test_compile.py", line 671 in test_stack_overflow
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/case.py", line 615 in run
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/case.py", line 663 in __call__
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 122 in run
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/suite.py", line 84 in __call__
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/unittest/runner.py", line 176 in run
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/support/__init__.py", line 1882 in _run_suite
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/support/__init__.py", line 1972 in run_unittest
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 179 in runtest_inner
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 286 in rerun_failed_tests
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 570 in _main
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 531 in main
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/libregrtest/main.py", line 584 in main
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/test/__main__.py", line 2 in <module>
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/runpy.py", line 85 in _run_code
  File "/home/buildbot/buildarea/3.7.einat-ubuntu/build/Lib/runpy.py", line 193 in _run_module_as_main
make: *** [buildbottest] Error 1
test_stack_overflow (test.test_compile.TestSpecifics) ... Makefile:1096: recipe for target 'buildbottest' failed
program finished with exit code 2
elapsedTime=6544.321095
History
Date User Action Args
2021-07-13 19:52:14pablogsalsetstatus: open -> closed
resolution: out of date
stage: resolved
2018-08-02 10:14:37pablogsalsetmessages: + msg322939
2018-08-02 10:03:37pablogsalsetmessages: + msg322935
2018-07-31 12:48:40pablogsalsettitle: test_compile is hanging im AMD Ubuntu buildbots -> test_compile hangs in AMD Ubuntu buildbots
2018-07-31 12:48:23pablogsalcreate