Author hniksic
Recipients hniksic
Date 2019-05-03.09:28:03
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1556875683.94.0.285155490659.issue36780@roundup.psfhosted.org>
In-reply-to
Content
At interpreter shutdown, Python waits for all pending futures of all executors to finish. There seems to be no way to disable the wait for pools that have been explicitly shut down with pool.shutdown(wait=False). The attached script demonstrates the issue.

In our code the futures are running blocking network calls that can be canceled by the user. The cancel action automatically cancels the pending futures and informs the running ones that they should exit. The remaining futures are those whose callables are "stuck" in network calls with long or infinite timeouts, such as reading from a non-responding network filesystem. Since those can't be interrupted, we use pool.shutdown(wait=False) to disown the whole pool. This works nicely, until the application exit, at which point it blocks trying to wait for the pending futures to finish. This can take an arbitrary amount of time, possibly never finishing.

A similar question has come up on StackOverflow, with the only answer recommending to unregister concurrent.futures.thread._python_exit: https://stackoverflow.com/q/48350257/1600898 . We are ourselves using a similar hack, but we would like to contribute a proper way to disown an executor.

The current behavior is explicitly documented, so presumably it can't be (easily) changed, but we could add a "disown" keyword argument to shutdown(), or a new disown() method, which would serve to explicitly disable the waiting. If this is considered desirable, I will create a pull request.
History
Date User Action Args
2019-05-03 09:28:03hniksicsetrecipients: + hniksic
2019-05-03 09:28:03hniksicsetmessageid: <1556875683.94.0.285155490659.issue36780@roundup.psfhosted.org>
2019-05-03 09:28:03hniksiclinkissue36780 messages
2019-05-03 09:28:03hniksiccreate