classification
Title: test_asyncio leaks dangling threads
Type: resource usage Stage: resolved
Components: asyncio, Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2017-08-21 16:33 by vstinner, last changed 2017-09-13 09:56 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3252 merged vstinner, 2017-08-31 13:39
PR 3264 merged vstinner, 2017-09-01 15:55
PR 3517 merged vstinner, 2017-09-12 17:30
Messages (7)
msg300637 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-21 16:33
Using PR 3138 of bpo-31234, I noticed that test_asyncio leaks dangling threads.

Tests which randomly leak threads:
---
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_no_inet_pton
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_service_name
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_connection_bluetooth
test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests.test_create_datagram_endpoint_no_addrinfo
test.test_asyncio.test_futures.DuckTests.test_wrap_future
test.test_asyncio.test_futures.CFutureTests.test_wrap_future
test.test_asyncio.test_futures.CFutureTests.test_wrap_future_use_global_loop
test.test_asyncio.test_futures.PyFutureTests.test_wrap_future
test.test_asyncio.test_futures.PyFutureTests.test_wrap_future_use_global_loop
test.test_asyncio.test_selector_events.BaseSelectorEventLoopTests.test_sock_connect_resolve_using_socket_params
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_with_timeout
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_task_cancelled
test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests.test_run_coroutine_threadsafe_task_factory_exception
---

Write this list into "tests" and run "python3 -m test --matchfile=tests".


haypo@selma$ ./python -u -m test -v test_asyncio --matchfile=tests
== CPython 3.7.0a0 (heads/concurrent_futures:f277fa3, Aug 21 2017, 18:11:41) [GCC 6.4.1 20170727 (Red Hat 6.4.1-1)]
(...)
test_create_connection_bluetooth (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... ok
test_create_connection_no_inet_pton (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 4 leaked threads (count: 4, dangling: 5)
ok
test_create_connection_service_name (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 4 leaked threads (count: 4, dangling: 5)
ok
test_create_datagram_endpoint_no_addrinfo (test.test_asyncio.test_base_events.BaseEventLoopWithSelectorTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future (test.test_asyncio.test_futures.CFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future_use_global_loop (test.test_asyncio.test_futures.CFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future (test.test_asyncio.test_futures.DuckTests) ... ok
test_wrap_future (test.test_asyncio.test_futures.PyFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_wrap_future_use_global_loop (test.test_asyncio.test_futures.PyFutureTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_sock_connect_resolve_using_socket_params (test.test_asyncio.test_selector_events.BaseSelectorEventLoopTests) ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_run_coroutine_threadsafe (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a thread to an event loop. ... ok
test_run_coroutine_threadsafe_task_cancelled (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a tread to an event loop ... ok
test_run_coroutine_threadsafe_task_factory_exception (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a tread to an event loop ... Warning -- threading_cleanup() detected 1 leaked threads (count: 1, dangling: 2)
ok
test_run_coroutine_threadsafe_with_timeout (test.test_asyncio.test_tasks.RunCoroutineThreadsafeTests)
Test coroutine submission from a thread to an event loop ... ok
(...)
msg301048 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-08-31 13:42
I proposed PR 3252 to only fix unit tests, but not asyncio.

An alternative is to modify BaseEventLoop.close() to replace executor.shutdown(wait=False) with executor.shutdown(wait=True), but it would change the behaviour. I'm not sure if we want to wait until all threads complete on close()?
msg301097 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-01 12:46
New changeset 16432beadb8eba079c9786cc0c0eaacfd9fd2f7b by Victor Stinner in branch 'master':
bpo-31250, test_asyncio: fix dangling threads (#3252)
https://github.com/python/cpython/commit/16432beadb8eba079c9786cc0c0eaacfd9fd2f7b
msg301099 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-01 12:52
I picked the safe option: only modify unit tests.
msg301131 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-01 15:55
Hum, EventLoopTestsMixin.tearDown() still has a bug. I reopen the issue.
msg301155 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-01 22:25
New changeset e8a533fbc734af6eeb389202ba6c6e9c2548027f by Victor Stinner in branch 'master':
bpo-31250, test_asyncio: fix EventLoopTestsMixin.tearDown() (#3264)
https://github.com/python/cpython/commit/e8a533fbc734af6eeb389202ba6c6e9c2548027f
msg301995 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-09-12 21:18
New changeset 5013a5ebc9978a58435036fa3860c465882c21da by Victor Stinner in branch '3.6':
[3.6] bpo-31250: test_asyncio: fix dangling threads (#3517)
https://github.com/python/cpython/commit/5013a5ebc9978a58435036fa3860c465882c21da
History
Date User Action Args
2017-09-13 09:56:51vstinnersetversions: + Python 3.6
2017-09-12 21:18:39vstinnersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2017-09-12 21:18:25vstinnersetmessages: + msg301995
2017-09-12 17:30:33vstinnersetkeywords: + patch
stage: resolved -> patch review
pull_requests: + pull_request3512
2017-09-01 22:25:42vstinnersetmessages: + msg301155
2017-09-01 15:55:46vstinnersetpull_requests: + pull_request3308
2017-09-01 15:55:41vstinnersetstatus: closed -> open
resolution: fixed -> (no value)
messages: + msg301131
2017-09-01 12:52:46vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg301099

stage: resolved
2017-09-01 12:46:09vstinnersetmessages: + msg301097
2017-08-31 13:42:00vstinnersetmessages: + msg301048
2017-08-31 13:39:56vstinnersetpull_requests: + pull_request3296
2017-08-21 16:33:56vstinnercreate