Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bytes-like objects with socket.sendall(), SSL, and http.client #71527

Closed
vadmium opened this issue Jun 17, 2016 · 4 comments
Closed

bytes-like objects with socket.sendall(), SSL, and http.client #71527

vadmium opened this issue Jun 17, 2016 · 4 comments
Labels
3.7 (EOL) end of life docs Documentation in the Doc dir stdlib Python modules in the Lib dir topic-SSL type-bug An unexpected behavior, bug, or error

Comments

@vadmium
Copy link
Member

vadmium commented Jun 17, 2016

BPO 27340
Nosy @tiran, @vadmium
PRs
  • bpo-27340: Use memoryview in SSLSocket.sendall() #3384
  • [3.6] bpo-27340: Use memoryview in SSLSocket.sendall() (GH-3384) #3434
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-09-08.00:05:19.262>
created_at = <Date 2016-06-17.04:25:20.448>
labels = ['3.7', 'expert-SSL', 'type-bug', 'library', 'docs']
title = 'bytes-like objects with socket.sendall(), SSL, and http.client'
updated_at = <Date 2017-09-08.00:05:19.261>
user = 'https://github.com/vadmium'

    bugs.python.org fields:

    activity = <Date 2017-09-08.00:05:19.261>
actor = 'christian.heimes'
assignee = 'docs@python'
closed = True
closed_date = <Date 2017-09-08.00:05:19.262>
closer = 'christian.heimes'
components = ['Documentation', 'Library (Lib)', 'SSL']
creation = <Date 2016-06-17.04:25:20.448>
creator = 'martin.panter'
dependencies = []
files = []
hgrepos = []
issue_num = 27340
keywords = ['patch']
message_count = 4.0
messages = ['268707', '301639', '301652', '301653']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'docs@python', 'martin.panter']
pr_nums = ['3384', '3434']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27340'
versions = ['Python 3.6', 'Python 3.7']
    @vadmium
    Copy link
    Member Author

    vadmium commented Jun 17, 2016

    According to the documentation, HTTPSConnection.request() should accept arbitrary bytes-like objects, but this is not the case. Currently (since bpo-23756), a “bytes-like object” is defined to be anything that works with Python’s buffer API, as long as it is C-contiguous. These objects can be passed to socket.sendall():

    >> byteslike = (ctypes.c_ubyte * 6).from_buffer_copy(b"DATA\r\n")
    >> s = socket.create_connection(("localhost", 80))
    >> s.sendall(byteslike) # Server receives b"DATA\r\n"

    This is not explicitly documented for socket objects. But since bpo-23539 (3.4+), HTTPConnection.request() does document support for bytes-like objects:

    >> h = HTTPConnection("localhost", 80)
    >> # Send Content-Length: 6 and body b"DATA\r\n"
    >> h.request("POST", "/", body=byteslike)

    On its own, there is no problem with Python relying on its own undocumented behaviour. But Python’s “ssl” module does not support arbitrary bytes-like objects, and as a result neither does HTTPSConnection:

    >>> s = ssl.wrap_socket(socket.create_connection(("localhost", 443)))
>>> s.sendall(byteslike)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'
>>> c = ssl.create_default_context(cafile="/lib/python3.5/test/keycert.pem")
>>> h = HTTPSConnection("localhost", 443, context=c)
>>> h.request("POST", "/", body=byteslike)
Traceback (most recent call last):
  File "/usr/lib/python3.5/http/client.py", line 885, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.5/ssl.py", line 886, in sendall
    v = self.send(data[count:])
  File "/usr/lib/python3.5/ssl.py", line 856, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.5/ssl.py", line 581, in write
    return self._sslobj.write(data)
TypeError: a bytes-like object is required, not 'list'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 913, in _send_output
    self.send(message_body)
  File "/usr/lib/python3.5/http/client.py", line 892, in send
    "or an iterable, got %r" % type(data))
TypeError: data should be a bytes-like object or an iterable, got <class '__main__.c_ubyte_Array_6'>

    This could be fixed in the implementation of SSLSocket.sendall(). But I am not sure if that is an appropriate change for 3.5. Another option would be to adjust the documentation of HTTP(S)Connection in 3.5, either not mentioning bytes-like objects at all, or clarifying that they don’t work with SSL.

    @vadmium vadmium added stdlib Python modules in the Lib dir docs Documentation in the Doc dir labels Jun 17, 2016
    @tiran tiran added 3.7 (EOL) end of life topic-SSL type-bug An unexpected behavior, bug, or error labels Sep 15, 2016
    @tiran
    Copy link
    Member

    tiran commented Sep 7, 2017

    New changeset 888bbdc by Christian Heimes in branch 'master':
    bpo-27340: Use memoryview in SSLSocket.sendall() (bpo-3384)
    888bbdc

    @tiran
    Copy link
    Member

    tiran commented Sep 7, 2017

    New changeset 9423f5d by Christian Heimes in branch '3.6':
    [3.6] bpo-27340: Use memoryview in SSLSocket.sendall() (GH-3384) (bpo-3434)
    9423f5d

    @tiran
    Copy link
    Member

    tiran commented Sep 8, 2017

    In Python 3.6 and master, SSLSocket.sendall() now accepts any bytes-like object that can be cast to bytes. Since it uses a sliced memoryview instead of sliced bytes, it avoids extra copies of the payload, too.

    @tiran tiran closed this as completed Sep 8, 2017
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life docs Documentation in the Doc dir stdlib Python modules in the Lib dir topic-SSL type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @tiran @vadmium