Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test.regrtest: add an option to run test.bisect_cmd on failed tests, use it on Refleaks buildbots #84184

Closed
vstinner opened this issue Mar 18, 2020 · 3 comments
Closed

test.regrtest: add an option to run test.bisect_cmd on failed tests, use it on Refleaks buildbots #84184

vstinner opened this issue Mar 18, 2020 · 3 comments
Labels
3.9 only security fixes tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 40003
Nosy @vstinner, @corona10, @shihai1991
PRs
  • bpo-40003: test.bisect_cmd copies Python options #19246
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2020-05-18.15:51:14.255>
created_at = <Date 2020-03-18.15:48:15.705>
labels = ['tests', '3.9']
title = 'test.regrtest: add an option to run test.bisect_cmd on failed tests, use it on Refleaks buildbots'
updated_at = <Date 2020-05-18.15:51:14.254>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2020-05-18.15:51:14.254>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-05-18.15:51:14.255>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-03-18.15:48:15.705>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40003
keywords = ['patch']
message_count = 3.0
messages = ['364528', '365392', '369256']
nosy_count = 3.0
nosy_names = ['vstinner', 'corona10', 'shihai1991']
pr_nums = ['19246']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40003'
versions = ['Python 3.9']
    @vstinner
    Copy link
    Member Author

    There are some tests which fail randomly in general, but fail in a deterministic way on some specific buildbot workers.

    bpo-39932 is a good example: test_multiprocessing_fork fails with "test_multiprocessing_fork leaked [0, 2, 0] file descriptors". The test fails while run in paralle, but it also fails when re-run sequentially. Except that when I connect to the buildbot worker, it does not fail anymore.

    test_multiprocessing_fork contains 356 test methods, the test file (Lib/test/_test_multiprocessing.py) has 5741 lines of Python code, and the multiprocessing is made of 8149 lines of Python code and 1133 lines of C code. It's hard to audit such code. The multiprocessing uses multiple proceses, pipes, signals, etc. It's really hard to debug.

    I propose to add an --bisect-failed option to test.regrtest to run test.bisect_cmd on failed tests. We can start to experiment it on Refleaks buildbots. Regular tests (not Refleaks tests) are easier to reproduce in general.

    It should speedup analysis of reference leak and "altered environment" test failures. Having less test methods to audit is way simpler.

    The implement should be that at the end of regrtest, after tests are re-run, run each failed test in test.bisect_cmd with the same command line arguments than test.regrtest.

    test.bisect_cmd uses 100 iterations by default. It's ok if the bisection fails to reduce the number of test methods. At least, it should reduce the list in some cases.

    @vstinner vstinner added 3.9 only security fixes tests Tests in the Lib/test dir labels Mar 18, 2020
    @vstinner
    Copy link
    Member Author

    New changeset 01e743d by Victor Stinner in branch 'master':
    bpo-40003: test.bisect_cmd copies Python options (GH-19246)
    01e743d

    @vstinner
    Copy link
    Member Author

    While it would be nice, I'm not sure if it's a good idea to make buildbot jobs even longer. I wrote this issue to investigate an issue that I fixed in the meanwhile, so I close the issue.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner