classification
Title: High level API for loop.run_in_executor(None, ...)?
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: antoine.pietri, asvetlov, chris.jerdonek, cjrh, gvanrossum, primal, yselivanov
Priority: normal Keywords:

Created on 2019-09-28 14:43 by antoine.pietri, last changed 2020-09-06 23:55 by gvanrossum. This issue is now closed.

Messages (12)
msg353456 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2019-09-28 14:43
In 3.7 a lot of high level functions have been added to the asyncio API, but nothing to start blocking functions as non-blocking threads. You still have to call get_event_loop() then await loop.run_in_executor(None, callable). I think this pattern is *very* common and could use a higher level interface.

Would an API like this make sense?

async def create_thread(callable, *args, *, kwargs=None, loop=None, timeout=None)

Then it could just be used like this:

await asyncio.create_thread(my_blocking_read, 4096, timeout=10)

This API could wrap the run_in_executor in an asyncio.Task, that way
you wouldn't have to await it to start the thread. There's evidence
that this has confused people in the past:

https://stackoverflow.com/questions/54263558/is-asyncio-run-in-executor-specified-ambiguously
msg353460 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2019-09-28 15:10
Actually I don't think it's possible with the current implementation to cancel the concurrent.Future after a timeout, so maybe we should remove that argument. So, this signature instead:

async def create_thread(callable, *args, *, kwargs=None, loop=None)
msg353483 - (view) Author: Paul Martin (primal) * Date: 2019-09-29 03:54
run_in_executor doesn't necessarily create a new thread each time so create_thread would be misleading. run_in_thread might be better.
msg353491 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-09-29 07:45
I'd like to see how thread pools integrate with (not existing yet bug planned for 3.9) task groups before pushing forward the proposed approach.
msg353648 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2019-10-01 08:26
> run_in_executor doesn't necessarily create a new thread each time so create_thread would be misleading. run_in_thread might be better.

Right, the idea was to have an analogy with create_task, because the run_in_executor would be wrapped in a Task. I'm okay with run_thread() too.
msg354529 - (view) Author: Caleb Hattingh (cjrh) * Date: 2019-10-12 11:24
Even before task groups land, this API can be easily improved by adding

asyncio.run_in_executor(func, *args, **kwargs)

- Only valid inside a coro or async context (uses get_running_loop internally)
- Analogous to how `loop.create_task` became `asyncio.create_task`
- Drop having to specify `None` for the default executor
- Users already know the `run_in_executor` name
- Allow both positional and kwargs (we can partial internally before calling loop.run_in_executor)
msg354530 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2019-10-12 11:39
On Sat, Oct 12, 2019 at 1:24 PM Caleb Hattingh <report@bugs.python.org> wrote:
> Even before task groups land, this API can be easily improved by adding
>
> asyncio.run_in_executor(func, *args, **kwargs)
>
> - Only valid inside a coro or async context (uses get_running_loop internally)
> - Analogous to how `loop.create_task` became `asyncio.create_task`
> - Drop having to specify `None` for the default executor
> - Users already know the `run_in_executor` name
> - Allow both positional and kwargs (we can partial internally before calling loop.run_in_executor)

I think it should be run_thread() if it only works for thread.
run_in_executor() should take an executor= parameter, but it wouldn't
be as clear for beginners. I think there's value in having a simple,
explicit way of saying "I want to run this function in a thread"
without having to know what executors are.
msg354531 - (view) Author: Caleb Hattingh (cjrh) * Date: 2019-10-12 11:53
We can't allow both an `executor=` kwarg, as well as **kwargs for the target func, unfortunately. If we do `executor=`, we'll again have to force users to use functools.partial to wrap their functions that take kwargs.
msg354532 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2019-10-12 12:18
Yes, that's why I'm in favor of calling it run_thread. Because it
fundamentally *isn't* a way to run a function in an executor, it's a
way to run a function in a thread.
msg369313 - (view) Author: Chris Jerdonek (chris.jerdonek) * (Python committer) Date: 2020-05-19 01:13
Is this issue the same as this one? https://bugs.python.org/issue32309
msg374166 - (view) Author: Antoine Pietri (antoine.pietri) * Date: 2020-07-24 09:15
Yeah I'm pretty sure this solves it: https://github.com/python/cpython/pull/20143

Closing the issue, thanks.
msg376472 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-09-06 23:55
Really closing.
History
Date User Action Args
2020-09-06 23:55:48gvanrossumsetstatus: open -> closed

nosy: + gvanrossum
messages: + msg376472

stage: resolved
2020-07-24 09:15:38antoine.pietrisetresolution: fixed
messages: + msg374166
2020-05-19 01:13:58chris.jerdoneksetnosy: + chris.jerdonek
messages: + msg369313
2019-10-12 12:18:20antoine.pietrisetmessages: + msg354532
2019-10-12 11:53:14cjrhsetmessages: + msg354531
2019-10-12 11:39:44antoine.pietrisetmessages: + msg354530
2019-10-12 11:24:06cjrhsetnosy: + cjrh
messages: + msg354529
2019-10-01 08:26:27antoine.pietrisetmessages: + msg353648
2019-09-29 07:45:37asvetlovsetmessages: + msg353491
2019-09-29 03:54:42primalsetnosy: + primal
messages: + msg353483
2019-09-28 15:10:25antoine.pietrisetmessages: + msg353460
2019-09-28 14:43:05antoine.pietricreate