classification
Title: asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads
Type: Stage:
Components: asyncio, Tests Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, vstinner, yselivanov
Priority: normal Keywords:

Created on 2018-07-03 22:57 by vstinner, last changed 2018-09-24 15:58 by vstinner.

Messages (5)
msg321006 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-03 22:57
AMD64 FreeBSD 10.x Shared 3.7:
http://buildbot.python.org/all/#/builders/124/builds/410

...
test_remove_fds_after_closing (test.test_asyncio.test_events.KqueueEventLoopTests) ... ok
test_run_in_executor (test.test_asyncio.test_events.KqueueEventLoopTests) ... ok
test_run_in_executor_cancel (test.test_asyncio.test_events.KqueueEventLoopTests) ...

  Warning -- threading_cleanup() failed to cleanup 1 threads (count: 1, dangling: 2)
  Dangling thread: <Thread(ThreadPoolExecutor-13_0, started daemon 34471091200)>
  Dangling thread: <_MainThread(MainThread, started 34393318400)>

ok
test_run_until_complete (test.test_asyncio.test_events.KqueueEventLoopTests) ... ok
test_run_until_complete_nesting (test.test_asyncio.test_events.KqueueEventLoopTests) ... ok
...
1 test altered the execution environment:
    test_asyncio
msg321534 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-07-12 10:20
It's easy to reproduce the issue on Linux:

diff --git a/Lib/test/test_asyncio/test_events.py b/Lib/test/test_asyncio/test_events.py
index 11cd950df1..df4c2b9849 100644
--- a/Lib/test/test_asyncio/test_events.py
+++ b/Lib/test/test_asyncio/test_events.py
@@ -359,7 +359,7 @@ class EventLoopTestsMixin:
             called = True
 
         def run():
-            time.sleep(0.05)
+            time.sleep(1.0)
 
         f2 = self.loop.run_in_executor(None, run)
         f2.cancel()


The problem is that BaseEventLoop.close() shutdowns its default executor  without waiting:

    def close(self):
        ...
        executor = self._default_executor
        if executor is not None:
            self._default_executor = None
            executor.shutdown(wait=True)

I fixed a similar issue in socketserver:

* bpo-31233: for socketserver.ThreadingMixIn
* bpo-31151: for socketserver.ForkingMixIn 
* bpo-33540: add block_on_close attr to socketserver

I suggest to wait by default, but maybe also add a block_on_close attribute to BaseEventLoop (default: False) just for backward compatibility.

What do you think Yury, Andrew, and Guido?
msg326050 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-21 22:19
Yury, Andrew: Do you know if the executor doesn't wait on purpose? Would it be possible to change that in Python 3.8?
msg326052 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-09-21 22:29
> Yury, Andrew: Do you know if the executor doesn't wait on purpose? Would it be possible to change that in Python 3.8?

Maybe.  At least we need to add a "timeout" argument to asyncio.run() to let it wait for executor jobs.

I'm also thinking about making OS threads cancellable/interruptable in Python 3.8 (if they run pure Python code).
msg326259 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-09-24 15:58
> Maybe.  At least we need to add a "timeout" argument to asyncio.run() to let it wait for executor jobs.

The shutdown() method of concurrent.futures.Executor API doesn't accept a timeout. It waits for multiple things.

I added "block_on_close = True" class attribute to socketserver.ForkingMixIn and socketserver.ThreadingMixIn. By default, server_close() waits until all children complete, but the wait is non-blocking if block_on_close is false.
History
Date User Action Args
2018-09-24 15:58:45vstinnersetmessages: + msg326259
2018-09-21 22:29:20yselivanovsetmessages: + msg326052
2018-09-21 22:19:51vstinnersetmessages: + msg326050
2018-07-13 21:46:20gvanrossumsetnosy: - gvanrossum
2018-07-12 10:21:26vstinnersettitle: test_asyncio: test_run_in_executor_cancel() leaked a dangling thread on AMD64 FreeBSD 10.x Shared 3.7 -> asyncio: BaseEventLoop.close() shutdowns the executor without waiting causing leak of dangling threads
2018-07-12 10:20:53vstinnersetnosy: + gvanrossum
messages: + msg321534
2018-07-03 22:57:26vstinnercreate