classification
Title: Configurable blocksize in HTTP(S)Connection
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brett.cannon, nirs, serhiy.storchaka, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2017-11-04 22:32 by nirs, last changed 2017-11-06 21:21 by vstinner. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 4279 merged nirs, 2017-11-04 22:34
Messages (5)
msg305571 - (view) Author: Nir Soffer (nirs) * Date: 2017-11-04 22:32
blocksize is hardcoded to 8192 in send() and _read_readable(), preventing
efficient upload when using file-like body.

Users of the module that are not interested in chunked encoding can rewrite
the copy loop using HTTPConnection.send():

   conn = HTTPSConnection(...)
   conn.putrequest(...)
   conn.putheader(...)
   conn.endheaders()

   while True:
       chunk = file.read(512*1024)
       if not chunk:
          break
       conn.send(chunk)

But fixing send() to use a configurable blocksize seems more useful.

Also, users of requests do not have access the underlying connection, so
they cannot use preferred buffer size.

When reading from /dev/zero and uploading to server that drop the received
data, larger buffer size gives 3X more throughput *and* 1/3 of cpu time.
With real storage and network, the effect will probably be much smaller.
msg305624 - (view) Author: Nir Soffer (nirs) * Date: 2017-11-06 09:57
When using highlevel request() api, users can control the block size by
wrapping the file object with an iterator:

    class FileIter:

        def __init__(self, file, blocksize):
            self.file = file
            self.blocksize = blocksize

        def __iter__(self):
            while True:
                datablock = self.file.read(self.blocksize)
                if not datablock:
                    break
                yield datablock

Adding configurable block size will avoid this workaround.
msg305678 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:16
New changeset ad455cd9243319b896c86074ffeb3bf78a82f4ec by Victor Stinner (Nir Soffer) in branch 'master':
bpo-31945: Configurable blocksize in HTTP(S)Connection (#4279)
https://github.com/python/cpython/commit/ad455cd9243319b896c86074ffeb3bf78a82f4ec
msg305679 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:18
Thank you Nir Soffer for this nice enhancement. Sadly, since it's a new feature, it cannot be backport to Python 3.6 nor 2.7, since it's a new feature.
msg305680 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2017-11-06 21:21
The commit message contains much more information than the NEWS and What's New entries. Maybe the What's New entry can be completed, I don't know. In the meanwhile, I close the issue :-) Nir: Feel free to create a new PR if you want to complete the What's New entry ;-)
History
Date User Action Args
2017-11-06 21:21:57vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg305680

stage: patch review -> resolved
2017-11-06 21:18:07vstinnersetmessages: + msg305679
2017-11-06 21:16:39vstinnersetmessages: + msg305678
2017-11-06 21:15:53vstinnersettype: enhancement
versions: - Python 2.7, Python 3.6, Python 3.8
2017-11-06 09:57:49nirssetmessages: + msg305624
2017-11-05 23:28:20martin.panterlinkissue21790 superseder
2017-11-04 22:34:44nirssetkeywords: + patch
stage: patch review
pull_requests: + pull_request4241
2017-11-04 22:32:30nirscreate