classification
Title: asyncio.create_subprocess_exec() only works in main thread
Type: behavior Stage: resolved
Components: asyncio Versions:
process
Status: closed Resolution: duplicate
Dependencies: Superseder: asyncio.create_subprocess_exec() only works with main event loop
View: 35621
Assigned To: Nosy List: asvetlov, stefan, vstinner, yselivanov
Priority: normal Keywords:

Created on 2019-01-02 02:07 by stefan, last changed 2019-06-02 11:01 by asvetlov. This issue is now closed.

Messages (8)
msg332855 - (view) Author: Stefan Seefeld (stefan) * (Python committer) Date: 2019-01-02 02:07
This is an addendum to issue35621:

To be able to call `asyncio.create_subprocess_exec()` from another thread, A separate event loop needs to be created. To make the child watcher aware of this new loop, I have to call `asyncio.get_child_watcher().attach_loop(loop)`. However, in the current implementation this call needs to be made by the main thread (or else the `signal` module will complain as handlers may only be registered in the main thread).

So, to work around the above limitations, the following workflow needs to be used:

1) create a new loop in the main thread
2) attach it to the child watcher
3) spawn a worker thread
4) set the previously created event loop as default loop

After that, I can run `asyncio.create_subprocess_exec()` in the worker thread. However, I suppose the worker thread will be the only thread able to call that function, given the child watcher's limitation to a single loop.

Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact.
msg333009 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 22:27
> Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact.

The current documentation says:

"To handle signals and to execute subprocesses, the event loop must be run in the main thread."

https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading

But I agree that the doc can be enhanced :-) Do you have suggestions?
msg333010 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-01-04 22:28
By the way, asyncio doc is outdated:
"The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details."
https://docs.python.org/dev/library/asyncio-subprocess.html#creating-subprocesses

It's no longer true in Python 3.8:

"Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop."
https://docs.python.org/dev/library/asyncio-platforms.html
msg333032 - (view) Author: Stefan Seefeld (stefan) * (Python committer) Date: 2019-01-05 02:14
That's quite an unfortunate limitation ! I'm working on a GUI frontend to a Python tool I wrote using asyncio, and the GUI (Qt-based) itself insists to run its own event loop in the main thread.

I'm not sure how to work around this limitation, but I can report that my previously reported strategy appears to be working well (so far).

What are the issues I should expect to encounter running an asyncio event loop in a worker thread ?
msg333191 - (view) Author: Stefan Seefeld (stefan) * (Python committer) Date: 2019-01-08 00:48
OK, so while I have been able to work around the issues (by using `quamash` to bridge between `asyncio` and `Qt`), I'd still like to understand the rationale behind the limitation that any subprocess-managing event-loop has to run in the main thread. Is this really an architectural limitation or a limit of the current implementation ?

And to your question: As I wasn't really expecting such a limitation, I would have expected 
    "To handle signals and to execute subprocesses, the event loop must be run in the main thread."

to be written much more prominently (as a warning admonition even, perhaps).
msg333195 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-01-08 01:27
The limitation is a consequence of how Linux works.
Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal.

On the other hand signal handlers in Python should work in the main thread.

Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it.
The behavior can be broken in next releases, sorry.

IIRC we discussed this scenario recently, official support for transferring a loop between threads is a too strict limitation, not all third-party implementations are ready for that.
msg333199 - (view) Author: Stefan Seefeld (stefan) * (Python committer) Date: 2019-01-08 01:54
> The limitation is a consequence of how Linux works.
> Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal.

Why does the `wait()` have to be non-blocking ? We can call it once in 
response to the reception of a `SIGCHILD`, where we know the call 
wouldn't block. Then we can pass the `pid` to whatever event loop 
created the subprocess to do the cleanup there...

> On the other hand signal handlers in Python should work in the main thread.

That's fine.

> Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it.
> The behavior can be broken in next releases, sorry.

Yeah, I observed some strange issues that looked like they could be 
fixed by someone intimately familiar with `asyncio`. But given the 
documented limitation, it seemed wise not to descend into that rabbit 
hole, and so I (at least temporarily) abandoned the entire approach.

--

Stefan

       ...ich hab' noch einen Koffer in Berlin...
msg344272 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2019-06-02 11:01
Fixed by #35621
History
Date User Action Args
2019-06-02 11:01:50asvetlovsetstatus: open -> closed
resolution: duplicate
messages: + msg344272

superseder: asyncio.create_subprocess_exec() only works with main event loop
stage: resolved
2019-01-08 01:54:28stefansetmessages: + msg333199
2019-01-08 01:27:38asvetlovsetmessages: + msg333195
2019-01-08 00:48:32stefansetmessages: + msg333191
2019-01-05 02:14:10stefansetmessages: + msg333032
2019-01-04 22:28:46vstinnersetmessages: + msg333010
2019-01-04 22:27:15vstinnersetnosy: + vstinner
messages: + msg333009
2019-01-02 02:07:20stefancreate