classification
Title: regrtest: re-run failed tests in a subprocess
Type: Stage: resolved
Components: Tests Versions: Python 3.8
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords:

Created on 2018-05-29 11:41 by vstinner, last changed 2018-09-19 23:56 by vstinner. This issue is now closed.

Messages (2)
msg318011 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 11:41
The -w option of regrtest (python -m test) re-run failed tests in verbose in the current process. Problem: if the test does crash, regrtest doesn't re-run following tests, it doesn't write the final tests result (like failing tests, total duration, etc.).

To make regrtest tolerant to such crashes, re-run tests should be run in verbose mode. The only constraint is to disable buffering on stdout and stderr to make sure that stdout and stderr order is kept in pipes. Or maybe stderr should be redirected to stdout to avoid ordering issue?
msg318013 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-29 11:42
A random example of such crash, process killed by faulthandler timeout:
http://buildbot.python.org/all/#/builders/118/builds/65

(...)
394 tests OK.

10 slowest tests:
- test_subprocess: 7 min 13 sec
- test_concurrent_futures: 2 min 47 sec
- test_multiprocessing_spawn: 2 min 29 sec
- test_urllib2net: 1 min 45 sec
- test_multiprocessing_forkserver: 1 min 34 sec
- test_nntplib: 1 min 29 sec
- test_smtpnet: 1 min 23 sec
- test_multiprocessing_fork: 1 min 22 sec
- test_tools: 1 min 19 sec
- test_tokenize: 1 min 13 sec

1 test failed:
    test_asyncio

20 tests skipped:
    test_dbm_gnu test_devpoll test_epoll test_gdb test_idle test_ioctl
    test_msilib test_ossaudiodev test_spwd test_startfile test_tcl
    test_tix test_tk test_ttk_guionly test_ttk_textonly test_turtle
    test_winconsoleio test_winreg test_winsound test_zipfile64
Re-running failed tests in verbose mode
Re-running test 'test_asyncio' in verbose mode
(...)
test_sock_client_ops (test.test_asyncio.test_events.PollEventLoopTests) ... ok

Timeout (0:15:00)!
Thread 0x000000080139a000 (most recent call first):
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/selectors.py", line 415 in select
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 1713 in _run_once
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 523 in run_forever
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/asyncio/base_events.py", line 555 in run_until_complete
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2135 in run_loop
  File "/usr/home/buildbot/python/3.7.koobs-freebsd-current/build/Lib/test/test_asyncio/test_events.py", line 2249 in test_sock_sendfile_mix_with_regular_send
  (...)

test_sock_sendfile_mix_with_regular_send (test.test_asyncio.test_events.PollEventLoopTests) ... *** Error code 1
Stop.
make: stopped in /usr/home/buildbot/python/3.7.koobs-freebsd-current/build
program finished with exit code 1
elapsedTime=2612.384935
History
Date User Action Args
2018-09-19 23:56:46vstinnersetstatus: open -> closed
resolution: out of date
stage: resolved
2018-05-29 11:42:42vstinnersetmessages: + msg318013
2018-05-29 11:41:03vstinnercreate