classification
Title: http.client.HTTPResponse raises IncompleteRead on chunked encoding
Type: Stage:
Components: Library (Lib) Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arden
Priority: normal Keywords:

Created on 2020-01-17 16:21 by Arden, last changed 2020-01-17 16:29 by Arden.

Messages (2)
msg360199 - (view) Author: Arden (Arden) Date: 2020-01-17 16:21
http.client.HTTPResponse._readinto_chunked has two problems in python 3.8 - 3.9.

1. _safe_readinto assumes that self.fp.readinto(b) will read exactly len(b) bytes. This is not always true, especially in case of SSL traffic. But _safe_readinto raises IncompleteRead if less len(b) bytes was read.

So, _safe_readinto should be removed and substituted with self.fp.readinto

2. _readinto_chunked may lose chunked block boundary because of this line:
  self.chunk_left = 0
it should be changed to
  self.chunk_left = chunk_left - n
in order to self._get_chunk_left() be able to find real chunk boundary

Corrected function looks like this:
def _readinto_chunked(self, b):
    assert self.chunked != _UNKNOWN
    total_bytes = 0
    mvb = memoryview(b)
    try:
        while True:
            chunk_left = self._get_chunk_left()
            if chunk_left is None:
                return total_bytes

            if len(mvb) <= chunk_left:
                n = self.fp.readinto(mvb)
                self.chunk_left = chunk_left - n
                return total_bytes + n

            temp_mvb = mvb[:chunk_left]
            n = self.fp.readinto(temp_mvb)
            mvb = mvb[n:]
            total_bytes += n
            self.chunk_left = chunk_left - n

    except IncompleteRead:
        raise IncompleteRead(bytes(b[0:total_bytes]))
msg360200 - (view) Author: Arden (Arden) Date: 2020-01-17 16:29
Pythons 3.5-3.7 also have
  self.chunk_left = 0
in  _readinto_chunked

I think it's bug but I didn't check it thoroughly
History
Date User Action Args
2020-01-17 16:29:52Ardensetmessages: + msg360200
2020-01-17 16:21:17Ardencreate