classification
Title: Add os.preadv() and os.pwritev()
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: YoSTEALTH, miss-islington, ned.deily, nitishch, njs, pitrou, vstinner
Priority: high Keywords: patch

Created on 2017-09-06 19:57 by YoSTEALTH, last changed 2018-05-30 23:26 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5239 merged pablogsal, 2018-01-19 01:55
PR 5415 closed vstinner, 2018-01-29 08:19
PR 5467 closed YoSTEALTH, 2018-02-01 03:28
PR 7254 merged pablogsal, 2018-05-30 21:10
PR 7258 merged miss-islington, 2018-05-30 23:01
Messages (18)
msg301508 - (view) Author: (YoSTEALTH) * Date: 2017-09-06 19:57
Asynchronous, Non-blocking Buffered File Read

"RWF_NONBLOCK" flag for os.open()

Link:
https://lwn.net/Articles/612483/
https://lwn.net/Articles/613068/
https://lwn.net/Articles/636967/
msg309898 - (view) Author: (YoSTEALTH) * Date: 2018-01-13 20:23
preadv2(2) syscall with RWF_NONBLOCK feature is now released starting linux4.14
https://kernelnewbies.org/Linux_4.14#Asynchronous_buffered_I.2FO_support
msg310003 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-15 17:26
It seems to me there are two issues here:

1. What are the pros and cons of adding support for this new mode, e.g. are there any potential gotchas?

2. Assuming the result of 1 is we should go ahead, then omeone needs to generate a pull request including configure checks as necessary, doc changes and tests.

It's getting late in the game for inclusion in 3.7 with only 2 weeks left until feature freeze.
msg310030 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-01-16 00:40
In any case, I think this would take the form of new low-level calls in the posix module, i.e. add thin wrappers around the new system / libc calls.
msg310033 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-01-16 02:09
Whoa, cool.

bpo-32561 is complementary to this. (They both make sense on their own, but they're even better together.)
msg310089 - (view) Author: (YoSTEALTH) * Date: 2018-01-16 13:37
According to this nginx test https://www.nginx.com/blog/thread-pools-boost-performance-9x/ there is a huge boost in performance when using thread poll for File IO. It is postulated that when preadv2() & RWF_NONBLOCK feature are available it would further benefit (latency, thread sync, ...)

I am not a C coder so i can't just jump into this and code this feature (if i could i would have done it long ago). As Antoine mentioned, it needs to written at C (+ctypes) level.

I am willing to help, hit me up on #python user: stealth_
msg310454 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-23 00:53
Explanation for myself.

https://kernelnewbies.org/Linux_4.14#Asynchronous_buffered_I.2FO_support

"In this release, the preadv2(2) syscall with RWF_NONBLOCK will let userspace applications bypass enqueuing operation in the threadpool if it's already available in the pagecache."

For applications using a thread pool, like the aiofiles does for asyncio, preadv2() allows to bypass the thread pool which is obviously faster. If the read block, just uses thread pool as currently done.

Same rationale for pwritev2().
msg310828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-27 01:35
I set the priority to release blocker to make sure that I don't forget to merge the PR before Python 3.7b1 (next monday?). The PR is *almost* ready, remaining issues are mostly related to documentation and coding style.
msg310829 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-27 01:39
Antoine Pitrou: "In any case, I think this would take the form of new low-level calls in the posix module, i.e. add thin wrappers around the new system / libc calls."

"new system / libc calls"... well, preadv() is available on FreeBSD since FreeBSD 6 :-) And preadv() and pwritev() first appeared in Linux 2.6.30; library support was added in glibc 2.10. It's not really something "new".

It seems like the two functions became more interesting for Python since Linux implemented the new RWF_NONBLOCK flag. aiofiles can be used to bypass its thread-pool, for better performances.
msg310831 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-27 01:53
FYI in 2013 I proposed to use writev() in the io module, but it was inefficient: bpo-17655.
msg310869 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-27 16:16
New changeset 4defba3b95ec0f52ce75b8466831d30fb5d333f3 by Victor Stinner (Pablo Galindo) in branch 'master':
bpo-31368: Expose preadv and pwritev in the os module (#5239)
https://github.com/python/cpython/commit/4defba3b95ec0f52ce75b8466831d30fb5d333f3
msg310881 - (view) Author: (YoSTEALTH) * Date: 2018-01-27 19:22
Thank you Pablo Galindo for your amazingly hard work. Also Victor Stinner for your guidance, thanks you :)
msg311020 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-28 22:29
If I understand the discussion on the PR properly, Victor would like to revisit the documentation for this PR but is OK with the code as merged. I'm therefore downgrading this to "deferred blocker".  Please try to get the issue closed one way or another prior to 3.7.0b2, thanks!
msg317536 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 06:28
As far as I can tell, this issue is still open only because of the original desire to expand the original documentation.  YoSTEALTH produced PR 5447 based on Victor's original doc update PR.  In the meantime, the current 3.7 docs have been updated somewhat.  YoSTEALTH, if you would be willing to update PR 5467 perhaps we can get it reviewed and merged and close this issue. In any case, I'm lowering the priority from "deferred blocker".
msg318158 - (view) Author: (YoSTEALTH) * Date: 2018-05-30 07:49
I can't at the moment, i am out of country for couple more months.
msg318218 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 23:00
New changeset 02e2a085dc1740b1cbf4ba2ed77335c84ce8a367 by Victor Stinner (Pablo Galindo) in branch 'master':
bpo-31368: Enhance os.preadv() documentation (GH-7254)
https://github.com/python/cpython/commit/02e2a085dc1740b1cbf4ba2ed77335c84ce8a367
msg318221 - (view) Author: miss-islington (miss-islington) Date: 2018-05-30 23:22
New changeset 0e823c6efa4729f3cd19f96af82c673b10cd3ee2 by Miss Islington (bot) in branch '3.7':
bpo-31368: Enhance os.preadv() documentation (GH-7254)
https://github.com/python/cpython/commit/0e823c6efa4729f3cd19f96af82c673b10cd3ee2
msg318223 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-05-30 23:26
Thanks Pablo Galindo for the new better documentation! (And thanks myself since I wrote the original documentation PR ;-))
History
Date User Action Args
2018-05-30 23:26:01vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg318223

stage: patch review -> resolved
2018-05-30 23:22:15miss-islingtonsetnosy: + miss-islington
messages: + msg318221
2018-05-30 23:01:25miss-islingtonsetpull_requests: + pull_request6885
2018-05-30 23:00:07vstinnersetmessages: + msg318218
2018-05-30 21:10:08pablogsalsetpull_requests: + pull_request6883
2018-05-30 07:49:22YoSTEALTHsetmessages: + msg318158
2018-05-24 06:28:12ned.deilysetpriority: deferred blocker -> high

messages: + msg317536
2018-02-01 03:28:31YoSTEALTHsetpull_requests: + pull_request5293
2018-01-29 08:19:46vstinnersetpull_requests: + pull_request5249
2018-01-28 22:29:43ned.deilysetpriority: release blocker -> deferred blocker

messages: + msg311020
versions: + Python 3.8
2018-01-27 19:22:50YoSTEALTHsetmessages: + msg310881
2018-01-27 16:16:39vstinnersetmessages: + msg310869
2018-01-27 01:53:17vstinnersetmessages: + msg310831
2018-01-27 01:47:06vstinnersettitle: Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK) -> Add os.preadv() and os.pwritev()
2018-01-27 01:39:04vstinnersetmessages: + msg310829
2018-01-27 01:35:37vstinnersetpriority: normal -> release blocker

messages: + msg310828
2018-01-23 00:53:31vstinnersetnosy: + vstinner
messages: + msg310454
2018-01-19 01:55:30pablogsalsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request5085
2018-01-16 13:37:58YoSTEALTHsetmessages: + msg310089
2018-01-16 07:10:04nitishchsetnosy: + nitishch
2018-01-16 02:09:07njssetmessages: + msg310033
2018-01-16 00:40:26pitrousetnosy: + njs, pitrou
messages: + msg310030
2018-01-15 17:26:42ned.deilysettitle: RWF_NONBLOCK -> Support asynchronous, non-blocking buffered reads (RWF_NONBLOCK)
nosy: + ned.deily

messages: + msg310003

components: + Library (Lib)
stage: needs patch
2018-01-13 20:23:40YoSTEALTHsetmessages: + msg309898
2017-09-06 19:57:29YoSTEALTHcreate