classification
Title: bytes-like objects with socket.sendall(), SSL, and http.client
Type: behavior Stage: resolved
Components: Documentation, Library (Lib), SSL Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: christian.heimes, docs@python, martin.panter
Priority: normal Keywords: patch

Created on 2016-06-17 04:25 by martin.panter, last changed 2017-09-08 00:05 by christian.heimes. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 3384 merged christian.heimes, 2017-09-06 13:55
PR 3434 merged christian.heimes, 2017-09-07 22:26
Messages (4)
msg268707 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-06-17 04:25
According to the documentation, HTTPSConnection.request() should accept arbitrary bytes-like objects, but this is not the case. Currently (since Issue 23756), a “bytes-like object” is defined to be anything that works with Python’s buffer API, as long as it is C-contiguous. These objects can be passed to socket.sendall():

>>> byteslike = (ctypes.c_ubyte * 6).from_buffer_copy(b"DATA\r\n")
>>> s = socket.create_connection(("localhost", 80))
>>> s.sendall(byteslike)  # Server receives b"DATA\r\n"

This is not explicitly documented for socket objects. But since Issue 23539 (3.4+), HTTPConnection.request() does document support for bytes-like objects:

>>> h = HTTPConnection("localhost", 80)
>>> # Send Content-Length: 6 and body b"DATA\r\n"
>>> h.request("POST", "/", body=byteslike)

On its own, there is no problem with Python relying on its own undocumented behaviour. But Python’s “ssl” module does not support arbitrary bytes-like objects, and as a result neither does HTTPSConnection:

>>> s = ssl.wrap_socket(socket.create_connection(("localhost", 443)))
>>> s.sendall(byteslike)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'
>>> c = ssl.create_default_context(cafile="/lib/python3.5/test/keycert.pem")
>>> h = HTTPSConnection("localhost", 443, context=c)
>>> h.request("POST", "/", body=byteslike)
Traceback (most recent call last):
  File "/usr/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 892, in send
    "or an iterable, got %r" % type(data))
TypeError: data should be a bytes-like object or an iterable, got <class '__main__.c_ubyte_Array_6'>

This could be fixed in the implementation of SSLSocket.sendall(). But I am not sure if that is an appropriate change for 3.5. Another option would be to adjust the documentation of HTTP(S)Connection in 3.5, either not mentioning bytes-like objects at all, or clarifying that they don’t work with SSL.
msg301639 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-07 22:26
New changeset 888bbdc192ec4db888a294ef758cf5510442dc9a by Christian Heimes in branch 'master':
bpo-27340: Use memoryview in SSLSocket.sendall() (#3384)
https://github.com/python/cpython/commit/888bbdc192ec4db888a294ef758cf5510442dc9a
msg301652 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-07 23:59
New changeset 9423f5d68874ff2e500cbe072c25f883cf754be8 by Christian Heimes in branch '3.6':
[3.6] bpo-27340: Use memoryview in SSLSocket.sendall() (GH-3384) (#3434)
https://github.com/python/cpython/commit/9423f5d68874ff2e500cbe072c25f883cf754be8
msg301653 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-08 00:05
In Python 3.6 and master, SSLSocket.sendall() now accepts any bytes-like object that can be cast to bytes. Since it uses a sliced memoryview instead of sliced bytes, it avoids extra copies of the payload, too.
History
Date User Action Args
2017-09-08 00:05:19christian.heimessetstatus: open -> closed
versions: - Python 3.5
messages: + msg301653

resolution: fixed
stage: patch review -> resolved
2017-09-07 23:59:24christian.heimessetmessages: + msg301652
2017-09-07 22:26:37christian.heimessetnosy: + christian.heimes
messages: + msg301639
2017-09-07 22:26:25christian.heimessetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request3431
2017-09-06 13:55:48christian.heimessetpull_requests: + pull_request3392
2016-09-15 08:09:03christian.heimessetstage: needs patch
type: behavior
components: + SSL
versions: + Python 3.7
2016-06-17 04:25:59martin.panterlinkissue23740 dependencies
2016-06-17 04:25:20martin.pantercreate