classification
Title: Use sendfile where possible in httplib
Type: performance Stage: needs patch
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: 17552 Superseder:
Assigned To: Nosy List: benjamin.peterson, eric.araujo, giampaolo.rodola, kasun, orsenthil, rosslagerwall
Priority: normal Keywords: easy, patch

Created on 2011-12-08 20:47 by benjamin.peterson, last changed 2014-06-11 12:57 by giampaolo.rodola.

Files
File name Uploaded Description Edit
httplib-sendfile.patch giampaolo.rodola, 2014-06-11 12:57 review
Messages (6)
msg149052 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-12-08 20:47
HTTPConnection.send() should use os.sendfile when possible to avoid copying data into userspace and back.
msg149073 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2011-12-09 04:52
This is not possible for two reasons:

- on most POSIX systems, sendfile() works with mmap-like ("regular") files only, while HTTPConnection.send() accepts any file-like object as long as it provides a read() method

- after read()ing a chunk of data from the file and before send()ing it over the socket, the data can be subject to an intermediate conversion (datablock.encode("iso-8859-1")):
http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
...whereas sendfile() can only be used to send a binary file "as-is"

I think we can use sendfile() in ftplib.py though .
I'll open a ticket for that.
msg149075 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-12-09 06:08
2011/12/8 Giampaolo Rodola' <report@bugs.python.org>:
>
> Giampaolo Rodola' <g.rodola@gmail.com> added the comment:
>
> This is not possible for two reasons:
>
> - on most POSIX systems, sendfile() works with mmap-like ("regular") files only, while HTTPConnection.send() accepts any file-like object as long as it provides a read() method
>
> - after read()ing a chunk of data from the file and before send()ing it over the socket, the data can be subject to an intermediate conversion (datablock.encode("iso-8859-1")):
> http://hg.python.org/cpython/file/87c6be1e393a/Lib/http/client.py#l839
> ...whereas sendfile() can only be used to send a binary file "as-is"

I presume you could check for a binary mode, though? Also, you can
catch EINVAl on invalid fds.
msg149077 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2011-12-09 07:00
ftplib's sendfile support is not tracked as issue13559.
Considerations I made there should apply here as well.
msg149078 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2011-12-09 07:01
Ops! I meant issue13564.
msg220262 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2014-06-11 12:57
Patch in attachment uses the newly added socket.sendfile() method (issue 17552).
History
Date User Action Args
2014-06-11 12:57:54giampaolo.rodolasetfiles: + httplib-sendfile.patch
type: enhancement -> performance
messages: + msg220262

keywords: + patch
2014-04-28 10:08:25giampaolo.rodolasetdependencies: + Add a new socket.sendfile() method
versions: + Python 3.5, - Python 3.3
2011-12-11 17:20:05kasunsetnosy: + kasun
2011-12-10 16:33:15eric.araujosetnosy: + eric.araujo
2011-12-10 13:33:00rosslagerwallsetnosy: + rosslagerwall
2011-12-09 07:01:52giampaolo.rodolasetmessages: + msg149078
2011-12-09 07:00:54giampaolo.rodolasetmessages: + msg149077
2011-12-09 06:08:50benjamin.petersonsetmessages: + msg149075
2011-12-09 04:52:30giampaolo.rodolasetmessages: + msg149073
2011-12-08 20:48:32pitrousetnosy: + orsenthil, giampaolo.rodola
2011-12-08 20:47:25benjamin.petersoncreate