classification
Title: RuntimeError after closing loop that used run_in_executor
Type: Stage: resolved
Components: asyncio Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, hniksic, miss-islington, yselivanov
Priority: normal Keywords: patch

Created on 2018-05-12 07:53 by hniksic, last changed 2018-05-29 15:17 by yselivanov. This issue is now closed.

Files
File name Uploaded Description Edit
executor-cancel hniksic, 2018-05-12 07:53
Pull Requests
URL Status Linked Edit
PR 7171 merged yselivanov, 2018-05-28 19:56
PR 7178 merged miss-islington, 2018-05-28 21:11
PR 7179 merged miss-islington, 2018-05-28 21:12
Messages (6)
msg316420 - (view) Author: Hrvoje Nikšić (hniksic) Date: 2018-05-12 07:53
Looking at a StackOverflow question[1], I was unable to find a way to correctly close an event loop that uses run_in_executor() for long-running tasks.

The question author tried to implement the following scenario:

1. create some tasks that use run_in_executor
2. run asyncio.wait(tasks, return_when=FIRST_EXCEPTION)
3. cancel pending tasks, if any
4. close the loop and continue with non-async work

However, when there are pending tasks from wait(), a RuntimeError is raised some time after step #4. In the test programs, it happens while waiting for the program to finish. I have attached a minimal example to reproduce the issue.

The immediate cause is that a callback installed by wrap_future() notices that the underlying concurrent.futures.Future is done and calls loop.call_soon_threadsafe() to copy the result to the asyncio.Future. call_soon_threadsafe() fails when the loop is closed.

This would be reasonable behavior if not for the fact that the code explicitly cancelled the asyncio future, and awaited it to ensure that the cancellation took effect. While it is clear that asyncio cannot interrupt a function already running in an executor, it should probably detach the connection between the concurrent future and the asyncio future, to prevent this kind of error (and possibly other problems).

For example, the _call_check_cancel closure in _chain_future could remove (or disable) the done_callback installed on source after the call to source.cancel(). Since at that point we know that destination (asyncio.Future) is already canceled, there is no longer value in invoking the done callback for source (concurrent.futures.Future).


[1]
https://stackoverflow.com/q/50279522/1600898
msg317888 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-28 16:43
Hopefully asyncio.run() in Python 3.7 will handle this case correctly.
msg317889 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-28 16:52
Ah, I see, the callback that tracks the state of the wrapped concurrent.Future doesn't check that the loop is closed and its future has been cancelled. I think this is a bug.
msg317925 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-28 21:10
New changeset fdccfe09f0b10776645fdb04a0783d6864c32b21 by Yury Selivanov in branch 'master':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/fdccfe09f0b10776645fdb04a0783d6864c32b21
msg317943 - (view) Author: miss-islington (miss-islington) Date: 2018-05-28 22:50
New changeset 8d8b86116fae91570c26fa48974b54986fbd1b72 by Miss Islington (bot) in branch '3.7':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/8d8b86116fae91570c26fa48974b54986fbd1b72
msg317972 - (view) Author: miss-islington (miss-islington) Date: 2018-05-29 01:32
New changeset a6d6bd70ac95a0f7bbfe07d4e60b43afcec370d2 by Miss Islington (bot) in branch '3.6':
bpo-33469: RuntimeError after closing loop that used run_in_executor (GH-7171)
https://github.com/python/cpython/commit/a6d6bd70ac95a0f7bbfe07d4e60b43afcec370d2
History
Date User Action Args
2018-05-29 15:17:35yselivanovsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-05-29 01:32:19miss-islingtonsetmessages: + msg317972
2018-05-28 22:50:04miss-islingtonsetnosy: + miss-islington
messages: + msg317943
2018-05-28 21:12:31miss-islingtonsetpull_requests: + pull_request6812
2018-05-28 21:11:31miss-islingtonsetpull_requests: + pull_request6811
2018-05-28 21:10:22yselivanovsetmessages: + msg317925
2018-05-28 19:56:30yselivanovsetkeywords: + patch
stage: patch review
pull_requests: + pull_request6806
2018-05-28 16:52:06yselivanovsetmessages: + msg317889
2018-05-28 16:43:24yselivanovsetmessages: + msg317888
2018-05-14 17:57:51hniksicsetnosy: + asvetlov, yselivanov
components: + asyncio
2018-05-12 07:53:08hniksiccreate