classification
Title: test_peg_generator timed out (25 min) on x86 Gentoo Non-Debug with X 3.9
Type: Stage: resolved
Components: Tests Versions: Python 3.9
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: lys.nikolaou, pablogsal, vstinner, zach.ware
Priority: normal Keywords:

Created on 2020-09-07 21:50 by vstinner, last changed 2020-11-16 17:06 by vstinner. This issue is now closed.

Messages (4)
msg376535 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-07 21:50
On the slow x86 Gentoo Non-Debug with X 3.9 buildbot, test_peg_generator timed out after 25 min, twice.

https://buildbot.python.org/all/#/builders/51/builds/24

Either, we increase the buildbot timeouts, test_peg_parser is made faster, or the test is splitted into multiple test files.

This buildbot worker is slow, so it sounds like a good idea to increase timeouts.


0:00:00 load avg: 4.79 Run tests in parallel using 2 child processes (timeout: 25 min, worker timeout: 30 min)
(...)
0:46:55 load avg: 6.60 [263/425/1] test_peg_generator crashed (Exit code 1) -- running: test_lib2to3 (5 min 20 sec)
Timeout (0:25:00)!
Thread 0xb7bcd700 (most recent call first):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1873 in _try_wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1915 in _wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1185 in wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 76 in spawn
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 91 in compile_c_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 104 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 80 in build_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 83 in run_test
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 446 in test_soft_keywords_lookahead

(...)
1:09:00 load avg: 6.25 Re-running test_peg_generator in verbose mode

(...)

10 slowest tests:
- test_tokenize: 7 min 54 sec
- test_unparse: 7 min 16 sec
- test_concurrent_futures: 6 min 44 sec
- test_lib2to3: 5 min 42 sec
- test_multiprocessing_spawn: 5 min 15 sec
- test_asyncio: 5 min 10 sec
- test_regrtest: 2 min 53 sec
- test_multiprocessing_forkserver: 2 min 38 sec
- test_unicodedata: 2 min 32 sec
- test_capi: 2 min 27 sec

1 test failed:
    test_peg_generator

(...)

test_syntax_error_for_string (test.test_peg_generator.test_c_parser.TestCParser) ... ok
test_ternary_operator (test.test_peg_generator.test_c_parser.TestCParser) ... ok
test_with_stmt_with_paren (test.test_peg_generator.test_c_parser.TestCParser) ... Timeout (0:25:00)!
Thread 0xb7bec700 (most recent call first):
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1873 in _try_wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1915 in _wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/subprocess.py", line 1185 in wait
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/spawn.py", line 76 in spawn
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 910 in spawn
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/unixccompiler.py", line 117 in _compile
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/ccompiler.py", line 574 in compile
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 529 in build_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 474 in _build_extensions_serial
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 449 in build_extensions
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/distutils/command/build_ext.py", line 340 in run
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/build.py", line 91 in compile_c_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Tools/peg_generator/pegen/testutil.py", line 104 in generate_parser_c_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 80 in build_extension
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 83 in run_test
  File "/buildbot/buildarea/cpython/3.9.ware-gentoo-x86.nondebug/build/Lib/test/test_peg_generator/test_c_parser.py", line 343 in test_with_stmt_with_paren
  ...
msg376541 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2020-09-08 01:14
These tests are expensive because they have to compile one C extension per test but I don't see any way to make them faster than does not complicate the whole setup considerably.

I have tried to compile a shared object containing the common parts but it just saves 35% of the time and is a bit of a mess to keep that state around.

Making the timeout bigger can be a possibility, certainly, but this also means the builder will get stuck more time in race conditions or deadlocks :(
msg376557 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-09-08 09:19
> Making the timeout bigger can be a possibility, certainly, but this also means the builder will get stuck more time in race conditions or deadlocks :(

Race conditions and deadlocks are less likely than false alarms of builds failed because the timeout is too short.

So far, I'm only aware of this specific buildbot worker which is too slow for "(timeout: 25 min, worker timeout: 30 min)". So just increase the timeout to 40 min, no?
msg381123 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-16 17:06
https://buildbot.python.org/all/#/builders/51 is green for 2 months, I close the issue.
History
Date User Action Args
2020-11-16 17:06:46vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg381123

stage: resolved
2020-09-08 09:19:11vstinnersetmessages: + msg376557
2020-09-08 01:14:17pablogsalsetnosy: + lys.nikolaou
messages: + msg376541
2020-09-07 21:50:53vstinnercreate