Looking at a StackOverflow question[1], I was unable to find a way to correctly close an event loop that uses run_in_executor() for long-running tasks.

The question author tried to implement the following scenario:

1. create some tasks that use run_in_executor
2. run asyncio.wait(tasks, return_when=FIRST_EXCEPTION)
3. cancel pending tasks, if any
4. close the loop and continue with non-async work

However, when there are pending tasks from wait(), a RuntimeError is raised some time after step #4. In the test programs, it happens while waiting for the program to finish. I have attached a minimal example to reproduce the issue.

The immediate cause is that a callback installed by wrap_future() notices that the underlying concurrent.futures.Future is done and calls loop.call_soon_threadsafe() to copy the result to the asyncio.Future. call_soon_threadsafe() fails when the loop is closed.

This would be reasonable behavior if not for the fact that the code explicitly cancelled the asyncio future, and awaited it to ensure that the cancellation took effect. While it is clear that asyncio cannot interrupt a function already running in an executor, it should probably detach the connection between the concurrent future and the asyncio future, to prevent this kind of error (and possibly other problems).

For example, the _call_check_cancel closure in _chain_future could remove (or disable) the done_callback installed on source after the call to source.cancel(). Since at that point we know that destination (asyncio.Future) is already canceled, there is no longer value in invoking the done callback for source (concurrent.futures.Future).


[1]
https://stackoverflow.com/q/50279522/1600898

Hopefully asyncio.run() in Python 3.7 will handle this case correctly.

Ah, I see, the callback that tracks the state of the wrapped concurrent.Future doesn't check that the loop is closed and its future has been cancelled. I think this is a bug.

New changeset fdccfe09f0b10776645fdb04a0783d6864c32b21 by Yury Selivanov in branch 'master':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/fdccfe09f0b10776645fdb04a0783d6864c32b21

New changeset 8d8b86116fae91570c26fa48974b54986fbd1b72 by Miss Islington (bot) in branch '3.7':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/8d8b86116fae91570c26fa48974b54986fbd1b72

New changeset a6d6bd70ac95a0f7bbfe07d4e60b43afcec370d2 by Miss Islington (bot) in branch '3.6':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/a6d6bd70ac95a0f7bbfe07d4e60b43afcec370d2