Issue35635
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2019-01-02 02:07 by stefan, last changed 2022-04-11 14:59 by admin. This issue is now closed.
Messages (8) | |||
---|---|---|---|
msg332855 - (view) | Author: Stefan Seefeld (stefan) * | Date: 2019-01-02 02:07 | |
This is an addendum to issue35621: To be able to call `asyncio.create_subprocess_exec()` from another thread, A separate event loop needs to be created. To make the child watcher aware of this new loop, I have to call `asyncio.get_child_watcher().attach_loop(loop)`. However, in the current implementation this call needs to be made by the main thread (or else the `signal` module will complain as handlers may only be registered in the main thread). So, to work around the above limitations, the following workflow needs to be used: 1) create a new loop in the main thread 2) attach it to the child watcher 3) spawn a worker thread 4) set the previously created event loop as default loop After that, I can run `asyncio.create_subprocess_exec()` in the worker thread. However, I suppose the worker thread will be the only thread able to call that function, given the child watcher's limitation to a single loop. Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact. |
|||
msg333009 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 22:27 | |
> Am I missing something ? Given the complexity of this, I would expect this to be better documented in the sections explaining how `asyncio.subprocess` and `threading` interact. The current documentation says: "To handle signals and to execute subprocesses, the event loop must be run in the main thread." https://docs.python.org/dev/library/asyncio-dev.html#concurrency-and-multithreading But I agree that the doc can be enhanced :-) Do you have suggestions? |
|||
msg333010 - (view) | Author: STINNER Victor (vstinner) * | Date: 2019-01-04 22:28 | |
By the way, asyncio doc is outdated: "The default asyncio event loop implementation on Windows does not support subprocesses. Subprocesses are available for Windows if a ProactorEventLoop is used. See Subprocess Support on Windows for details." https://docs.python.org/dev/library/asyncio-subprocess.html#creating-subprocesses It's no longer true in Python 3.8: "Changed in version 3.8: On Windows, ProactorEventLoop is now the default event loop." https://docs.python.org/dev/library/asyncio-platforms.html |
|||
msg333032 - (view) | Author: Stefan Seefeld (stefan) * | Date: 2019-01-05 02:14 | |
That's quite an unfortunate limitation ! I'm working on a GUI frontend to a Python tool I wrote using asyncio, and the GUI (Qt-based) itself insists to run its own event loop in the main thread. I'm not sure how to work around this limitation, but I can report that my previously reported strategy appears to be working well (so far). What are the issues I should expect to encounter running an asyncio event loop in a worker thread ? |
|||
msg333191 - (view) | Author: Stefan Seefeld (stefan) * | Date: 2019-01-08 00:48 | |
OK, so while I have been able to work around the issues (by using `quamash` to bridge between `asyncio` and `Qt`), I'd still like to understand the rationale behind the limitation that any subprocess-managing event-loop has to run in the main thread. Is this really an architectural limitation or a limit of the current implementation ? And to your question: As I wasn't really expecting such a limitation, I would have expected "To handle signals and to execute subprocesses, the event loop must be run in the main thread." to be written much more prominently (as a warning admonition even, perhaps). |
|||
msg333195 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-01-08 01:27 | |
The limitation is a consequence of how Linux works. Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal. On the other hand signal handlers in Python should work in the main thread. Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it. The behavior can be broken in next releases, sorry. IIRC we discussed this scenario recently, official support for transferring a loop between threads is a too strict limitation, not all third-party implementations are ready for that. |
|||
msg333199 - (view) | Author: Stefan Seefeld (stefan) * | Date: 2019-01-08 01:54 | |
> The limitation is a consequence of how Linux works. > Unix has no cross-platform API for non-blocking waiting for child process finish except handling SIGCHILD signal. Why does the `wait()` have to be non-blocking ? We can call it once in response to the reception of a `SIGCHILD`, where we know the call wouldn't block. Then we can pass the `pid` to whatever event loop created the subprocess to do the cleanup there... > On the other hand signal handlers in Python should work in the main thread. That's fine. > Your trick with a loop creation in the main thread and actual running in another thread can work, but asyncio doesn't guarantee it. > The behavior can be broken in next releases, sorry. Yeah, I observed some strange issues that looked like they could be fixed by someone intimately familiar with `asyncio`. But given the documented limitation, it seemed wise not to descend into that rabbit hole, and so I (at least temporarily) abandoned the entire approach. -- Stefan ...ich hab' noch einen Koffer in Berlin... |
|||
msg344272 - (view) | Author: Andrew Svetlov (asvetlov) * | Date: 2019-06-02 11:01 | |
Fixed by #35621 |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:59:09 | admin | set | github: 79816 |
2019-06-02 11:01:50 | asvetlov | set | status: open -> closed resolution: duplicate messages: + msg344272 superseder: asyncio.create_subprocess_exec() only works with main event loop stage: resolved |
2019-01-08 01:54:28 | stefan | set | messages: + msg333199 |
2019-01-08 01:27:38 | asvetlov | set | messages: + msg333195 |
2019-01-08 00:48:32 | stefan | set | messages: + msg333191 |
2019-01-05 02:14:10 | stefan | set | messages: + msg333032 |
2019-01-04 22:28:46 | vstinner | set | messages: + msg333010 |
2019-01-04 22:27:15 | vstinner | set | nosy:
+ vstinner messages: + msg333009 |
2019-01-02 02:07:20 | stefan | create |