Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

regrtest: run tests in subprocesses with -j1 on buildbots #69472

Closed
vstinner opened this issue Sep 30, 2015 · 2 comments
Closed

regrtest: run tests in subprocesses with -j1 on buildbots #69472

vstinner opened this issue Sep 30, 2015 · 2 comments
Labels
tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 25285
Nosy @vstinner
Files
  • regrtest_run_in_subprocess.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2016-05-20.11:42:12.402>
created_at = <Date 2015-09-30.20:36:02.789>
labels = ['tests']
title = 'regrtest: run tests in subprocesses with -j1 on buildbots'
updated_at = <Date 2016-05-20.11:42:12.401>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2016-05-20.11:42:12.401>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-05-20.11:42:12.402>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2015-09-30.20:36:02.789>
creator = 'vstinner'
dependencies = []
files = ['40632']
hgrepos = []
issue_num = 25285
keywords = ['patch']
message_count = 2.0
messages = ['251962', '265936']
nosy_count = 2.0
nosy_names = ['vstinner', 'Arfrever']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue25285'
versions = ['Python 3.6']
    @vstinner
    Copy link
    Member Author

    Currently, regrtest ignores the -j1 parameter. For -jN, the multiprocess mode is only enabled for N != 1. I propose to modify regrtest to run tests in subprocesses when -j1 is used. It would solve the issue bpo-18906.

    I also want to run tests in subprocesses in buildbot to better isolate tests. Running all tests in the same process, sequentially, cause various issues:

    • various kinds of side effects
    • memory leaks
    • random failures (unreliable tests)

    "make buildbot" already runs regrtest with -j1. I propose to also modify Tools/buildbot/test.bat to use run tests with -j1 by default on Windows.

    I fixed regrtest to setup tests using the same code for classic mode and multiprocess mode: slaves (child processes running tests for regrtest -jN) now inherit
    --memlimit/-M, --threshold/-t and --nowindows/-n options.

    Only 3 remaining functions are incompatible with -jN: --coverage/-T and --findleaks/-l.

    Finally, the multiprocess mode of regrtest now displays the duration of tests which took longer than 30 seconds, and every minute, it displays the tests running since longer than 30 seconds. Example with multiple workers (-j4):
    ---
    ...
    [395/399/1] test_mimetypes -- running: test_lzma (111 sec), test_multiprocessing_forkserver (224 sec)
    [396/399/1] test_poplib -- running: test_lzma (111 sec), test_multiprocessing_forkserver (225 sec)
    [397/399/1] test_lzma (111 sec) -- running: test_multiprocessing_forkserver (225 sec)
    [398/399/1] test_range -- running: test_multiprocessing_forkserver (227 sec)
    running: test_multiprocessing_forkserver (287 sec)
    [399/399/1] test_multiprocessing_forkserver (340 sec)
    378 tests OK.
    ---

    Note: "python -m test -j1 -j4" uses 4 workers, only the last -jN parameter is used. Some buildbots add -jN to use more workers.

    @vstinner vstinner added the tests Tests in the Lib/test dir label Sep 30, 2015
    @vstinner
    Copy link
    Member Author

    changeset: 101449:6f7e5b51c8fb
    user: Victor Stinner <victor.stinner@gmail.com>
    date: Fri May 20 13:15:55 2016 +0200
    files: Lib/test/libregrtest/cmdline.py Misc/NEWS Tools/buildbot/test.bat
    description:
    regrtest doesn't ignore -j1 anymore

    • regrtest now uses subprocesses when the -j1 command line option
      is used: each test file runs in a fresh child process. Before, the -j1 option
      was ignored.
    • Tools/buildbot/test.bat script now uses -j1 by default to run
      each test file in fresh child process.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner