This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: asyncio: will shutdown_default_executor work in single step (stop, run_forever) mode?
Type: Stage:
Components: Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: aeros, cmeyer
Priority: normal Keywords:

Created on 2020-04-21 22:00 by cmeyer, last changed 2022-04-11 14:59 by admin.

Messages (2)
msg366945 - (view) Author: Chris Meyer (cmeyer) * Date: 2020-04-21 22:00
Is the new asyncio.loop.shutdown_default_executor() suitable for event loops that are run in single-step mode?

event_loop.create_task(event_loop.shutdown_default_executor())
event_loop.stop()
event_loop.run_forever()

I don't see how it will work since shutdown_default_executor() may not be finished during a single 'stopped' run_forever() call. 

Also, what happens to pending executor futures? Previously reported bug #28464.

Here is my currently working code for shutting down the event loop.

# give event loop one chance to finish up
event_loop.stop()
event_loop.run_forever()
# wait for everything to finish, including tasks running in executors
# this assumes that all outstanding tasks finish in a reasonable time (i.e. no infinite loops).
all_tasks_fn = getattr(asyncio, "all_tasks", None)
if not all_tasks_fn:
    all_tasks_fn = asyncio.Task.all_tasks
tasks = all_tasks_fn(loop=event_loop)
if tasks:
    gather_future = asyncio.gather(*tasks, return_exceptions=True)
else:
    # work around fact that gather always uses global event loop in Python 3.8
    gather_future = event_loop.create_future()
    gather_future.set_result([])
event_loop.run_until_complete(gather_future)
# due to a seeming bug in Python libraries, the default executor needs to be shutdown explicitly before the event loop
# see http://bugs.python.org/issue28464 .
_default_executor = getattr(event_loop, "_default_executor", None)
if _default_executor:
    _default_executor.shutdown()
event_loop.close()
msg368224 - (view) Author: Kyle Stanley (aeros) * (Python committer) Date: 2020-05-06 04:43
> Is the new asyncio.loop.shutdown_default_executor() suitable for event loops that are run in single-step mode?

I honestly can't say for certain; the primary intended use case for shutdown_default_executor() was to provide a means of properly finalizing the resources associated with the default executor without blocking the event loop, notably the threads in the ThreadPoolExecutor (which were intermittently left dangling).  So, when working on the implementation w/ Yury and Andrew, I did not strongly consider single-step event loops. I was more concerned with how it fit in with asyncio.run() and safe finalization of executor resources. See https://bugs.python.org/issue34037 for context.

If you have a recommendation for a change to the current version shutdown_default_executor() that would help provide compatibility with single-step event loops without hindering the primary goals, I'm sure it would be considered.

> Also, what happens to pending executor futures?

When using `loop.shutdown_default_executor()`, it calls executor.shutdown(wait=True), which waits for submitted futures to the executor to complete before joining the executor's workers (regardless of whether they're threads or processes). So, the executor should not be terminated prior to the pending futures being completed.

From a glance at the example code posted above, it seems like it would be incompatible with asyncio.run(), which is a requirement for shutdown_default_executor(). See https://github.com/python/cpython/blob/b9c46a2c2d7fc68457bff641f78932d66f5e5f59/Lib/asyncio/runners.py#L8.
History
Date User Action Args
2022-04-11 14:59:29adminsetgithub: 84537
2020-05-06 04:43:16aerossetmessages: + msg368224
2020-04-21 22:41:58vstinnersetnosy: - vstinner
2020-04-21 22:41:51vstinnersetnosy: + aeros
2020-04-21 22:00:47cmeyercreate