classification
Title: test_concurrent_futures leaks dangling threads
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner
Priority: normal Keywords: patch

Created on 2017-08-21 16:04 by vstinner, last changed 2017-09-13 00:45 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3167 merged vstinner, 2017-08-21 16:12
PR 3178 merged vstinner, 2017-08-22 10:41
PR 3253 merged vstinner, 2017-08-31 13:54
PR 3521 merged vstinner, 2017-09-12 23:49
PR 3522 merged python-dev, 2017-09-13 00:06
Messages (8)
msg300634 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-21 16:04
Using PR 3138 of bpo-31234, I noticed that test_concurrent_futures leaks dangling threads:

haypo@selma$ ./python -u -m test --fail-env-changed test_concurrent_futures -v -m ThreadPoolWaitTests
== CPython 3.7.0a0 (heads/master:0267128, Aug 21 2017, 17:23:11) [GCC 6.4.1 20170727 (Red Hat 6.4.1-1)]
(...)
test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
0.11s ok
test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.60s ok
test_first_completed_some_already_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.60s ok
test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
3.11s ok
test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 2.10s ok
test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadPoolWaitTests) ... Warning -- threading_cleanup() detected 0 leaked threads (count: 0, dangling: 6)
1.60s ok
test_pending_calls_race (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.12s ok
test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... 6.11s ok
(...)
msg300651 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-21 21:24
New changeset 489d91c61d02f401551966f1558ee2dc87fc0ad9 by Victor Stinner in branch 'master':
bpo-31249: test_concurrent_futures checks dangling threads (#3167)
https://github.com/python/cpython/commit/489d91c61d02f401551966f1558ee2dc87fc0ad9
msg300697 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-22 14:50
New changeset bc61315377056fe362b744d9c44e17cd3178ce54 by Victor Stinner in branch 'master':
bpo-31249: Fix ref cycle in ThreadPoolExecutor (#3178)
https://github.com/python/cpython/commit/bc61315377056fe362b744d9c44e17cd3178ce54
msg301098 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-01 12:46
New changeset 60f3f1fb5cbf5354c3081138be806c56cb04153f by Victor Stinner in branch '3.6':
bpo-31249: Fix ref cycle in ThreadPoolExecutor (#3253)
https://github.com/python/cpython/commit/60f3f1fb5cbf5354c3081138be806c56cb04153f
msg302007 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-12 23:33
I still see the warning sometimes:

0:04:09 load avg: 7.03 [185/405/1] test_concurrent_futures failed (env changed) (68 sec) -- running: test_multiprocessing_spawn (160 sec), test_gdb (53 sec), test_lib2to3 (195 sec)
test_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_cancelled (test.test_concurrent_futures.FutureTests) ... ok
test_done (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_cancelled (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_failed (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_already_successful (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_raises (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_exception (test.test_concurrent_futures.FutureTests) ... ok
test_done_callback_with_result (test.test_concurrent_futures.FutureTests) ... ok
test_exception_with_success (test.test_concurrent_futures.FutureTests) ... ok
test_exception_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
test_repr (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_cancel (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_success (test.test_concurrent_futures.FutureTests) ... ok
test_result_with_timeout (test.test_concurrent_futures.FutureTests) ... ok
test_running (test.test_concurrent_futures.FutureTests) ... ok
test_correct_timeout_exception_msg (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
test_duplicate_futures (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 2.16s ok
test_free_reference_yielded_future (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
test_no_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 0.15s ok
test_zero_timeout (test.test_concurrent_futures.ProcessPoolAsCompletedTests) ... 2.15s ok
test_free_reference (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
test_killed_child (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
test_map (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.18s ok
test_map_chunksize (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
test_map_exception (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.17s ok
test_map_timeout (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 6.16s ok
test_max_workers_negative (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
test_no_stale_references (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
test_submit (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.14s ok
test_submit_keyword (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.13s ok
test_traceback (test.test_concurrent_futures.ProcessPoolExecutorTest) ... 0.15s ok
test_context_manager_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.07s ok
test_del_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.05s Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_hang_issue12364 (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 1.09s ok
test_interpreter_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 2.67s ok
test_processes_terminate (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.05s ok
test_run_after_shutdown (test.test_concurrent_futures.ProcessPoolShutdownTest) ... 0.00s ok
test_all_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 0.16s ok
test_first_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.64s ok
test_first_completed_some_already_completed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.65s ok
test_first_exception (test.test_concurrent_futures.ProcessPoolWaitTests) ... 3.15s ok
test_first_exception_one_already_failed (test.test_concurrent_futures.ProcessPoolWaitTests) ... 2.15s ok
test_first_exception_some_already_complete (test.test_concurrent_futures.ProcessPoolWaitTests) ... 1.65s ok
test_timeout (test.test_concurrent_futures.ProcessPoolWaitTests) ... 6.15s ok
test_correct_timeout_exception_msg (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.16s ok
test_duplicate_futures (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 2.15s ok
test_free_reference_yielded_future (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.15s ok
test_no_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 0.16s ok
test_zero_timeout (test.test_concurrent_futures.ThreadPoolAsCompletedTests) ... 2.15s ok
test_default_workers (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.15s ok
test_free_reference (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
test_map (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.15s ok
test_map_exception (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.16s ok
test_map_submits_without_iteration (test.test_concurrent_futures.ThreadPoolExecutorTest)
Tests verifying issue 11777. ... 0.15s ok
test_map_timeout (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 6.13s ok
test_max_workers_negative (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
test_no_stale_references (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
test_shutdown_race_issue12456 (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.16s ok
test_submit (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
test_submit_keyword (test.test_concurrent_futures.ThreadPoolExecutorTest) ... 0.14s ok
test_context_manager_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.07s ok
test_del_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.04s ok
test_hang_issue12364 (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 1.07s ok
test_interpreter_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 1.65s ok
test_run_after_shutdown (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.00s ok
test_thread_names_assigned (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.05s ok
test_thread_names_default (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.04s ok
test_threads_terminate (test.test_concurrent_futures.ThreadPoolShutdownTest) ... 0.02s ok
test_all_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.14s ok
test_first_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.65s ok
test_first_completed_some_already_completed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.65s ok
test_first_exception (test.test_concurrent_futures.ThreadPoolWaitTests) ... 3.14s ok
test_first_exception_one_already_failed (test.test_concurrent_futures.ThreadPoolWaitTests) ... 2.16s ok
test_first_exception_some_already_complete (test.test_concurrent_futures.ThreadPoolWaitTests) ... 1.62s ok
test_pending_calls_race (test.test_concurrent_futures.ThreadPoolWaitTests) ... 0.16s ok
test_timeout (test.test_concurrent_futures.ThreadPoolWaitTests) ... 6.13s ok

----------------------------------------------------------------------
Ran 79 tests in 68.182s

OK
msg302011 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 00:05
New changeset 3bcf157c115ba3e48bce62ac8cb13c703475a113 by Victor Stinner in branch 'master':
bpo-31249: Fix test_concurrent_futures dangling thread (#3521)
https://github.com/python/cpython/commit/3bcf157c115ba3e48bce62ac8cb13c703475a113
msg302016 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 00:43
New changeset 94eb2d5b73d07bc447f1fe19923b4c3259734b9c by Victor Stinner (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-31249: Fix test_concurrent_futures dangling thread (GH-3521) (#3522)
https://github.com/python/cpython/commit/94eb2d5b73d07bc447f1fe19923b4c3259734b9c
msg302017 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-13 00:45
Ok, the issue should now be fixed in Python 3.6 and master (future 3.7).
History
Date User Action Args
2017-09-13 00:45:29vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg302017

stage: patch review -> resolved
2017-09-13 00:43:47vstinnersetmessages: + msg302016
2017-09-13 00:06:16python-devsetpull_requests: + pull_request3520
2017-09-13 00:05:55vstinnersetmessages: + msg302011
2017-09-12 23:49:26vstinnersetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request3519
2017-09-12 23:33:45vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg302007
2017-09-01 12:49:12vstinnersetstatus: open -> closed
resolution: fixed
stage: resolved
2017-09-01 12:46:46vstinnersetmessages: + msg301098
2017-08-31 13:54:53vstinnersetpull_requests: + pull_request3297
2017-08-22 14:50:45vstinnersetmessages: + msg300697
2017-08-22 13:58:19vstinnersetversions: + Python 3.6
2017-08-22 10:41:00vstinnersetpull_requests: + pull_request3216
2017-08-21 21:24:26vstinnersetmessages: + msg300651
2017-08-21 16:12:36vstinnersetpull_requests: + pull_request3203
2017-08-21 16:04:21vstinnersettype: resource usage
components: + Tests
2017-08-21 16:04:13vstinnercreate