Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use writev() function in the io module #61855

Closed
vstinner opened this issue Apr 8, 2013 · 5 comments
Closed

Use writev() function in the io module #61855

vstinner opened this issue Apr 8, 2013 · 5 comments
Labels
performance Performance or resource usage topic-IO

Comments

@vstinner
Copy link
Member

vstinner commented Apr 8, 2013

BPO 17655
Nosy @gpshead, @pitrou, @vstinner, @alex

Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-04-08.19:47:29.769>
created_at = <Date 2013-04-08.01:00:15.350>
labels = ['invalid', 'expert-IO', 'performance']
title = 'Use writev() function in the io module'
updated_at = <Date 2013-04-08.19:47:29.768>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2013-04-08.19:47:29.768>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2013-04-08.19:47:29.769>
closer = 'vstinner'
components = ['IO']
creation = <Date 2013-04-08.01:00:15.350>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 17655
keywords = []
message_count = 5.0
messages = ['186259', '186261', '186281', '186283', '186340']
nosy_count = 5.0
nosy_names = ['gregory.p.smith', 'pitrou', 'vstinner', 'alex', 'neologix']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue17655'
versions = ['Python 3.4']

@vstinner
Copy link
Member Author

vstinner commented Apr 8, 2013

Since issue bpo-12268 has been fixed, it looks like it became easier to modify the io to use the writev() function when available.

For example, if FileIO.writelines() uses writev(), it can be used by TextIOWrapper.write() through BufferedWriter. The _io.TextIOWrapper.write() method stores encoded chunks of text into a list. It can calls buffer.writlines(pending) instead of buffer.write(b''.join(pending)).

I expect less Python function calls and less system calls, and so better performances because both are expensive (especially I/O syscalls).

See also issue bpo-15723.

I don't know if/how readv() can be used to optimize I/O performances.

@vstinner vstinner added topic-IO performance Performance or resource usage labels Apr 8, 2013
@vstinner
Copy link
Member Author

vstinner commented Apr 8, 2013

@neologix
Copy link
Mannequin

neologix mannequin commented Apr 8, 2013

I somehow doubt that the gain is worth the trouble, vectored disk I/O is not as interesting as vectored read/writes to a NIC.

Actually, a quick search returned this link:
http://www.mail-archive.com/dev@httpd.apache.org/msg23763.html

Running the benchmark written by the Apache guys:
$ rm -f writev.out; sync; sleep 5; ./test
writev: 1s526601.
copy+write: 1s323405.

Doesn't really surprise me.

So I'm -1, since it's unlikely to yield any improvement, and will greatly complicate the code.

@pitrou
Copy link
Member

pitrou commented Apr 8, 2013

Agreed with Charles-François, it probably won't make a difference in practice.

@vstinner
Copy link
Member Author

vstinner commented Apr 8, 2013

I get similar Fedora 18 (Linux kernel 3.8.1), ext4 filesystem:

$ rm -f writev.out; sync; sleep 5; ./copy_write
copy+write: 1s576530.
$ rm -f writev.out; sync; sleep 5; ./writev 
writev: 1s686619.

I agree to close the issue. At least this issue can be used later is someone else ask why Python is not using writev() ;-)

@vstinner vstinner closed this as completed Apr 8, 2013
@ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
performance Performance or resource usage topic-IO
Projects
None yet
Development

No branches or pull requests

2 participants