classification
Title: Use writev() function in the io module
Type: performance Stage:
Components: IO Versions: Python 3.4
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: alex, gregory.p.smith, neologix, pitrou, vstinner
Priority: normal Keywords:

Created on 2013-04-08 01:00 by vstinner, last changed 2013-04-08 19:47 by vstinner. This issue is now closed.

Messages (5)
msg186259 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-04-08 01:00
Since issue #12268 has been fixed, it looks like it became easier to modify the io to use the writev() function when available.

For example, if FileIO.writelines() uses writev(), it can be used by TextIOWrapper.write() through BufferedWriter. The _io.TextIOWrapper.write() method stores encoded chunks of text into a list. It can calls buffer.writlines(pending) instead of buffer.write(b''.join(pending)).

I expect less Python function calls and less system calls, and so better performances because both are expensive (especially I/O syscalls).

See also issue #15723.

I don't know if/how readv() can be used to optimize I/O performances.
msg186261 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-04-08 01:02
Read also http://mail.python.org/pipermail/python-dev/2012-August/121396.html
msg186281 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-04-08 09:09
I somehow doubt that the gain is worth the trouble, vectored disk I/O is not as interesting as vectored read/writes to a NIC.

Actually, a quick search returned this link:
http://www.mail-archive.com/dev@httpd.apache.org/msg23763.html

Running the benchmark written by the Apache guys:
$ rm -f writev.out; sync; sleep 5; ./test
writev: 1s526601.
copy+write: 1s323405.

Doesn't really surprise me.

So I'm -1, since it's unlikely to yield any improvement, and will greatly complicate the code.
msg186283 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-04-08 09:58
Agreed with Charles-Fran├žois, it probably won't make a difference in practice.
msg186340 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-04-08 19:47
I get similar Fedora 18 (Linux kernel 3.8.1), ext4 filesystem:

$ rm -f writev.out; sync; sleep 5; ./copy_write
copy+write: 1s576530.
$ rm -f writev.out; sync; sleep 5; ./writev 
writev: 1s686619.

I agree to close the issue. At least this issue can be used later is someone else ask why Python is not using writev() ;-)
History
Date User Action Args
2013-04-08 19:47:29vstinnersetstatus: open -> closed
resolution: not a bug
messages: + msg186340
2013-04-08 09:58:20pitrousetmessages: + msg186283
2013-04-08 09:09:05neologixsetnosy: + neologix
messages: + msg186281
2013-04-08 01:02:48vstinnersetmessages: + msg186261
2013-04-08 01:02:43vstinnersetmessages: - msg186260
2013-04-08 01:02:33vstinnersetmessages: + msg186260
2013-04-08 01:01:10alexsetnosy: + alex
2013-04-08 01:00:15vstinnercreate