Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test_concurrent_futures leaks dangling threads #75432

Closed
vstinner opened this issue Aug 21, 2017 · 8 comments
Closed

test_concurrent_futures leaks dangling threads #75432

vstinner opened this issue Aug 21, 2017 · 8 comments
Labels
3.7 (EOL) end of life performance Performance or resource usage tests Tests in the Lib/test dir

Comments

@vstinner
Copy link
Member

BPO 31249
Nosy @vstinner
PRs
  • bpo-31249: test_concurrent_futures checks dangling threads #3167
  • bpo-31249: Fix ref cycle in ThreadPoolExecutor #3178
  • [3.6] bpo-31249: Fix ref cycle in ThreadPoolExecutor #3253
  • bpo-31249: Fix test_concurrent_futures dangling thread #3521
  • [3.6] bpo-31249: Fix test_concurrent_futures dangling thread (GH-3521) #3522
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-09-13.00:45:29.558>
created_at = <Date 2017-08-21.16:04:13.038>
labels = ['3.7', 'tests', 'performance']
title = 'test_concurrent_futures leaks dangling threads'
updated_at = <Date 2017-09-13.00:45:29.556>
user = 'https://github.com/vstinner'

    bugs.python.org fields:

    activity = <Date 2017-09-13.00:45:29.556>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-09-13.00:45:29.558>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2017-08-21.16:04:13.038>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 31249
keywords = ['patch']
message_count = 8.0
messages = ['300634', '300651', '300697', '301098', '302007', '302011', '302016', '302017']
nosy_count = 1.0
nosy_names = ['vstinner']
pr_nums = ['3167', '3178', '3253', '3521', '3522']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue31249'
versions = ['Python 3.6', 'Python 3.7']
    @vstinner
    Copy link
    Member Author

    Using PR 3138 of bpo-31234, I noticed that test_concurrent_futures leaks dangling threads:

    haypo@selma$ ./python -u -m test --fail-env-changed test_concurrent_futures -v -m ThreadPoolWaitTests
    == CPython 3.7.0a0 (heads/master:0267128, Aug 21 2017, 17:23:11) [GCC 6.4.1 20170727 (Red Hat 6.4.1-1)]
    (...)
    test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
    0.11s ok
    test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.60s ok
    test_first_completed_some_already_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.60s ok
    test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
    3.11s ok
    test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 2.10s ok
    test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
    1.60s ok
    test_pending_calls_race (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.12s ok
    test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... 6.11s ok
    (...)

    @vstinner vstinner added 3.7 (EOL) end of life tests Tests in the Lib/test dir performance Performance or resource usage labels Aug 21, 2017
    @vstinner
    Copy link
    Member Author

    New changeset 489d91c by Victor Stinner in branch 'master':
    bpo-31249: test_concurrent_futures checks dangling threads (bpo-3167)
    489d91c

    @vstinner
    Copy link
    Member Author

    New changeset bc61315 by Victor Stinner in branch 'master':
    bpo-31249: Fix ref cycle in ThreadPoolExecutor (bpo-3178)
    bc61315

    @vstinner
    Copy link
    Member Author

    vstinner commented Sep 1, 2017

    New changeset 60f3f1f by Victor Stinner in branch '3.6':
    bpo-31249: Fix ref cycle in ThreadPoolExecutor (bpo-3253)
    60f3f1f

    @vstinner vstinner closed this as completed Sep 1, 2017
    @vstinner
    Copy link
    Member Author

    I still see the warning sometimes:

    0:04:09 load avg: 7.03 [185/405/1] test_concurrent_futures failed (env changed) (68 sec) -- running: test_multiprocessing_spawn (160 sec), test_gdb (53 sec), test_lib2to3 (195 sec)
    test_cancel (test.test_concurrent_futures.FutureTests) ... ok
    test_cancelled (test.test_concurrent_futures.FutureTests) ... ok
    test_done (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_already_cancelled (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_already_failed (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_already_successful (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_raises (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_with_exception (test.test_concurrent_futures.FutureTests) ... ok
    test_done_callback_with_result (test.test_concurrent_futures.FutureTests) ... ok
    test_exception_with_success (test.test_concurrent_futures.FutureTests) ... ok
    test_exception_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
    test_repr (test.test_concurrent_futures.FutureTests) ... ok
    test_result_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
    test_result_with_success (test.test_concurrent_futures.FutureTests) ... ok
    test_result_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
    test_running (test.test_concurrent_futures.FutureTests) ... ok
    test_correct_timeout_exception_msg (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
    test_duplicate_futures (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 2.16s ok
    test_free_reference_yielded_future (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
    test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
    test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 2.15s ok
    test_free_reference (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
    test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
    test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.18s ok
    test_map_chunksize (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
    test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
    test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 6.16s ok
    test_max_workers_negative (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
    test_no_stale_references (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
    test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
    test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
    test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.13s ok
    test_traceback (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
    test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.07s ok
    test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.05s Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
    ok
    test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 1.09s ok
    test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 2.67s ok
    test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.05s ok
    test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.00s ok
    test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 0.16s ok
    test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.64s ok
    test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.65s ok
    test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... 3.15s ok
    test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 2.15s ok
    test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.65s ok
    test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... 6.15s ok
    test_correct_timeout_exception_msg (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.16s ok
    test_duplicate_futures (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 2.15s ok
    test_free_reference_yielded_future (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.15s ok
    test_no_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.16s ok
    test_zero_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 2.15s ok
    test_default_workers (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.15s ok
    test_free_reference (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
    test_map (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.15s ok
    test_map_exception (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.16s ok
    test_map_submits_without_iteration (test.test_concurrent_futures.ThreadPoolExecutorTest)
    Tests verifying bpo-11777. ... 0.15s ok
    test_map_timeout (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 6.13s ok
    test_max_workers_negative (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
    test_no_stale_references (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
    test_shutdown_race_issue12456 (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.16s ok
    test_submit (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
    test_submit_keyword (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
    test_context_manager_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.07s ok
    test_del_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.04s ok
    test_hang_issue12364 (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 1.07s ok
    test_interpreter_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 1.65s ok
    test_run_after_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.00s ok
    test_thread_names_assigned (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.05s ok
    test_thread_names_default (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.04s ok
    test_threads_terminate (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.02s ok
    test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.14s ok
    test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.65s ok
    test_first_completed_some_already_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.65s ok
    test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... 3.14s ok
    test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 2.16s ok
    test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.62s ok
    test_pending_calls_race (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.16s ok
    test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... 6.13s ok

    ----------------------------------------------------------------------
    Ran 79 tests in 68.182s

    OK

    @vstinner vstinner reopened this Sep 12, 2017
    @vstinner
    Copy link
    Member Author

    New changeset 3bcf157 by Victor Stinner in branch 'master':
    bpo-31249: Fix test_concurrent_futures dangling thread (bpo-3521)
    3bcf157

    @vstinner
    Copy link
    Member Author

    New changeset 94eb2d5 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
    [3.6] bpo-31249: Fix test_concurrent_futures dangling thread (GH-3521) (bpo-3522)
    94eb2d5

    @vstinner
    Copy link
    Member Author

    Ok, the issue should now be fixed in Python 3.6 and master (future 3.7).

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life performance Performance or resource usage tests Tests in the Lib/test dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @vstinner