classification
Title: regrtest: don't fail immediately if a child does crash
Type: enhancement Stage: resolved
Components: Tests Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: ammar2, python-dev, vstinner
Priority: normal Keywords: patch

Created on 2017-01-24 15:35 by vstinner, last changed 2017-02-06 13:13 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
regrtest_crash.diff ammar2, 2017-01-28 21:45 review
regrtest_crashed.patch vstinner, 2017-02-01 15:55 review
Messages (7)
msg286194 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-24 15:35
When regrtest is used with -jN, regrtest spawn child processes to run tests. If a child crashs, regrtest master process immedialtely fails. It would prefer to be able to continue to run following tests. It's not because a single test crashed that all tests would crash.

Example of output when a crash occurs:
-----------------
(...)
0:02:45 [118/404/3] test_bytes crashed -- running: test_multiprocessing_spawn (67 sec), test_io (54 sec), test_subprocess (76 sec)
python: Objects/abstract.c:2186: _Py_CheckFunctionResult: Assertion `(callable != NULL) ^ (where != NULL)' failed.
Fatal Python error: Aborted

Current thread 0x00007f9210f11440 (most recent call first):
  File "/home/haypo/prog/python/git_cpython/Lib/unittest/case.py", line 177 in handle
  File "/home/haypo/prog/python/git_cpython/Lib/unittest/case.py", line 728 in assertRaises
  File "/home/haypo/prog/python/git_cpython/Lib/test/test_bytes.py", line 636 in test_maketrans
  (...)
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 85 in _run_code
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 193 in _run_module_as_main

Traceback (most recent call last):
  File "/home/haypo/prog/python/git_cpython/Lib/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  (...)
  File "/home/haypo/prog/python/git_cpython/Lib/test/libregrtest/main.py", line 411, in run_tests
    run_tests_multiprocess(self)
  File "/home/haypo/prog/python/git_cpython/Lib/test/libregrtest/runtest_mp.py", line 221, in run_tests_multiprocess
    raise Exception(msg)
Exception: Child error on test_bytes: Exit code -6
-----------------
msg286195 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-01-24 15:36
At least, I would like to see which tests were executed, which tests failed, total duration, etc. Get a summary!
msg286419 - (view) Author: Ammar Askar (ammar2) * (Python triager) Date: 2017-01-28 21:45
This patch makes it so that when a child runner crashes during a multiprocess test run, it is treated similar to a keyboard interrupt. That is, the currently running tests will continue to run and immediately after it will print out a summary.

I think the test suite should be interrupted when an interpreter crash happens because this indicates a fairly major failure and interrupting the suite brings a lot of attention to it.
msg286639 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-01 15:55
Attached patch changes libregrtest to handle CHILD_ERROR as a normal result and continue the execution of following tests. As a result, it displays the test summary as usual on such crash.

--

Ammar Askar: Thanks for regrtest_crash.diff, but I prefer to execute other unit tests rather than exiting immediatly.

The --failfast option can be used with my patch to exit at the first crash or failure (with my patch, a crash is handled as a test failure).
msg287122 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-06 11:54
New changeset 4446613000a3 by Victor Stinner in branch 'default':
regrtest: don't fail immediately if a child does crash
https://hg.python.org/cpython/rev/4446613000a3
msg287123 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2017-02-06 12:00
New changeset 99e4e687145a76ac28055a651ee31470496c3ac7 by Victor Stinner in branch 'master':
regrtest: don't fail immediately if a child does crash
https://github.com/python/cpython/commit/99e4e687145a76ac28055a651ee31470496c3ac7
msg287124 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-02-06 13:13
I fixed regrtest in Python 3.7. Maybe I will backport the fix later to other branches, but I prefer to limit regrtest changes in stable branches.
History
Date User Action Args
2017-02-06 13:13:02vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg287124

stage: resolved
2017-02-06 12:00:25python-devsetmessages: + msg287123
2017-02-06 11:54:28python-devsetnosy: + python-dev
messages: + msg287122
2017-02-01 15:55:38vstinnersetfiles: + regrtest_crashed.patch

messages: + msg286639
2017-01-28 21:45:11ammar2setfiles: + regrtest_crash.diff

nosy: + ammar2
messages: + msg286419

keywords: + patch
2017-01-24 15:36:04vstinnersetmessages: + msg286195
2017-01-24 15:35:21vstinnercreate