This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: io_uring as a new backend to selectors and asyncio
Type: enhancement Stage: resolved
Components: asyncio Versions: Python 3.11
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Add support for io_uring to cpython
View: 41271
Assigned To: Nosy List: achimnol, asvetlov, corona10, njs, terry.reedy, yselivanov
Priority: normal Keywords:

Created on 2021-07-26 07:49 by achimnol, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (5)
msg398215 - (view) Author: Joongi Kim (achimnol) * Date: 2021-07-26 07:49
This is a rough early idea suggestion on adding io_uring as an alternative I/O multiplexing mechanism in Python (maybe selectors and asyncio). io_uring is a relatively new I/O mechanism introduced in Linux kernel 5.1 or later.

https://lwn.net/Articles/776703/
https://lwn.net/Articles/810414/
https://blogs.oracle.com/linux/post/an-introduction-to-the-io_uring-asynchronous-io-framework

The advantages of io_uring over epoll:
 - completion-based
 - less number of syscalls
 - higher performance (https://twitter.com/hielkedv/status/1218891982636027905)
 - file I/O support including read/write/stat/open/close

I'm not sure that io_uring would bring actual performance improvements to Python (and asyncio) or not yet.
We need some exploration and prototyping, but still technically it would be a nice-to-have feature, considering Python's recent speed-up optimizations.
Also io_uring is also intended to support high-speed storage devices such as NVMe, and may be a good addition to asyncio in terms of improved async file I/O support.

Here are existing attempts to incorporate uring in other languages:
 - liburing (C, https://github.com/axboe/liburing)
 - iou, tokio-uring (Rust, https://tokio.rs/blog/2021-07-tokio-uring)

I don't have any estimation on the efforts and time required to do the work,
but just want to spark the discussion. :)
msg398216 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2021-07-26 08:02
I think this is a dupe of issue41271?
msg398217 - (view) Author: Joongi Kim (achimnol) * Date: 2021-07-26 08:09
Ah, yes, but one year has passed so it may be another chance to discuss its adoption, as new advances like tokio_uring became available.
msg398218 - (view) Author: Joongi Kim (achimnol) * Date: 2021-07-26 08:18
As in the previous discussion, instead of tackling stdlib right away, it would be nice to evaluate the approach using 3rd-party libs, such as trio and/or async-tokio, or maybe a new library.

I have a strong feeling that we need to improve the async file I/O.
AFAIK, aiofiles is the only choice we have and it uses a thread pool, which involves many more context switches than required.
msg398607 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2021-07-30 20:28
Continue the discussion, including adding new info, to the #41271
History
Date User Action Args
2022-04-11 14:59:47adminsetgithub: 88901
2021-07-30 20:28:25terry.reedysetstatus: open -> closed

superseder: Add support for io_uring to cpython

nosy: + terry.reedy
messages: + msg398607
resolution: duplicate
stage: resolved
2021-07-26 08:18:22achimnolsetmessages: + msg398218
2021-07-26 08:09:51achimnolsetmessages: + msg398217
2021-07-26 08:02:22njssetmessages: + msg398216
2021-07-26 07:49:50achimnolcreate