Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Join started threads in tests #67987

Closed
serhiy-storchaka opened this issue Mar 28, 2015 · 4 comments
Closed

Join started threads in tests #67987

serhiy-storchaka opened this issue Mar 28, 2015 · 4 comments
Assignees
@serhiy-storchaka
Labels
tests Tests in the Lib/test dir type-feature A feature request or enhancement

Comments

@serhiy-storchaka
Copy link
Member

BPO 23799
Nosy @pitrou, @vstinner, @ezio-melotti, @voidspace, @serhiy-storchaka
Files
  • test_support_start_threads.patch
  • test_support_start_threads_2.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2015-04-04.06:53:34.628>
created_at = <Date 2015-03-28.20:10:30.324>
labels = ['type-feature', 'tests']
title = 'Join started threads in tests'
updated_at = <Date 2015-04-04.06:53:34.627>
user = 'https://github.com/serhiy-storchaka'

    bugs.python.org fields:

    activity = <Date 2015-04-04.06:53:34.627>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2015-04-04.06:53:34.628>
closer = 'serhiy.storchaka'
components = ['Tests']
creation = <Date 2015-03-28.20:10:30.324>
creator = 'serhiy.storchaka'
dependencies = []
files = ['38720', '38738']
hgrepos = []
issue_num = 23799
keywords = ['patch']
message_count = 4.0
messages = ['239465', '239583', '239601', '239780']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'ezio.melotti', 'michael.foord', 'python-dev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue23799'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']
    @serhiy-storchaka
    Copy link
    Member Author

    When a test starts many threads in a loop, there is a chance that the starting thread will fail due to lack of memory. In this case all started threads left dangling and make MemotyError even more probably. They also can provoke dim error messages in this and in following tests. Proposed patch adds new helper function test.support.start_threads() that starts threads and then join started threads.

    Some tests even hang without this patch.

    @serhiy-storchaka serhiy-storchaka added tests Tests in the Lib/test dir type-feature A feature request or enhancement labels Mar 28, 2015
    @vstinner
    Copy link
    Member

    In this case all started threads left dangling and make MemotyError even more probably

    Oh, I didn't know that. Good catch.

    test_support_start_threads.patch looks good to me and it looks like a nice enhancement.

    But would it be possible to emit a warning if a thread hangs?

    For example, try to join with a timeout of 60 seconds, and then display a warning, and retry with a timeout of 120 seconds, then display a second warning, etc.

    Maybe we may even give up after a timeout (ex: 5 minutes, or 15 minutes, it's up to you). An exception must be raised in this case. You can use faulthandler.dump_traceback() to display the traceback of all threads in this case.

    I hope that my concern is more theorical, but it would reduce the duration of tests when the test suite is stuck (which is quite common), and so being able to test more changesets per day.

    --

    I noticed that under low memory condition, some tests hang because Python doesn't handle notify the "parent thread" that the "child thread" fails to start. So the parent thread is stuck and nothing happens. Well, I'm not sure that it's related to this issue. At least, test_support_start_threads.patch doesn't help.

    @serhiy-storchaka
    Copy link
    Member Author

    Something like this?

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 1, 2015

    New changeset f8358f6e50e7 by Serhiy Storchaka in branch '2.7':
    Issue bpo-23799: Added test.test_support.start_threads() for running and
    https://hg.python.org/cpython/rev/f8358f6e50e7

    New changeset 8987218adc99 by Serhiy Storchaka in branch '3.4':
    Issue bpo-23799: Added test.support.start_threads() for running and cleaning up
    https://hg.python.org/cpython/rev/8987218adc99

    New changeset a2df4baa112b by Serhiy Storchaka in branch 'default':
    Issue bpo-23799: Added test.support.start_threads() for running and cleaning up
    https://hg.python.org/cpython/rev/a2df4baa112b

    @serhiy-storchaka serhiy-storchaka self-assigned this Apr 4, 2015
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees

    @serhiy-storchaka serhiy-storchaka

    Labels
    tests Tests in the Lib/test dir type-feature A feature request or enhancement
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @vstinner @serhiy-storchaka