classification
Title: httplib use wrong hostname in https request with SNI support
Type: Stage: test needed
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: lvhancy, martin.panter
Priority: normal Keywords:

Created on 2016-01-29 12:35 by lvhancy, last changed 2016-02-02 07:30 by lvhancy. This issue is now closed.

Messages (5)
msg259207 - (view) Author: lirenke (lvhancy) Date: 2016-01-29 12:35
httplib give openssl SNI extension message like IP:PORT string. 
the apache server would return 400 code if SNI/request ServerName mismatch.

In class HTTPSConnection, we hope call self._get_hostport() before give the value to server_hostname.

===
if self._tunnel_host:
	server_hostname = self._tunnel_host
else:
	server_hostname = self.host

self.sock = self._context.wrap_socket(self.sock,
									  server_hostname=server_hostname)

===
msg259307 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-01 02:28
The code fragment you posted looks like it is from HTTPSConnection.connect() <https://hg.python.org/cpython/file/v2.7.11/Lib/httplib.py#l1272>. But _get_hostport() is already called to set self.host in __init__(), and to set self._tunnel_host in set_tunnel(). So I do not understand what you are proposing. Can you provide a patch?

Failing that, can you give a demonstration where the SNI and “request ServerName” (is this the Host header field?) mismatch?

The only potential bug I can see is if you specify the host by IP address, the IP address is sent as the SNI, when RFC 6066 seems to say a literal IP address is not permitted.

Client (run in Python 2.7.11):
>>> conn = HTTPSConnection("127.0.0.1:44300", context=ssl._create_unverified_context())
>>> conn.request("GET", "/")

Server (run in Python 3.6):
>>> server = socket()
>>> server.bind(("localhost", 44300))
>>> server.listen()
>>> context = SSLContext(PROTOCOL_SSLv23)
>>> @context.set_servername_callback
... def callback(conn, name, context):
...     print(f"Requested server name {name!r}")
...     context = SSLContext(PROTOCOL_SSLv23)
...     context.load_cert_chain("Lib/test/keycert.pem")
...     conn.context = context
... 
>>> [conn, _] = server.accept()
>>> wrapped = context.wrap_socket(conn, server_side=True)
Requested server name '127.0.0.1'

My understanding is the client shouldn’t use SNI here, in which case the server name would be None.
msg259361 - (view) Author: lirenke (lvhancy) Date: 2016-02-02 05:41
In RFC6066, literal IPv4 is not allowed as hostname indeed. Actually, many requests still use the format of "IP+PORT" to access the server, and it seems Python don't prohibit this action explicitly. The explorer Chrome also use literal IP address to access for instance.

In our case, all requests will be forwarded by apacheproxy and there is another apache server that receiving them. The URL is like "https://128.6.42.21:8088/xx/", and the SNI will be added by OpenSSL in TLS-handshake packet when new https connection create. In this time, "128.6.42.21:8088" is set to self._tunnel_host in set_tunnel(), then, the server_hostname, as SNI, is determined.

The Server side's apache will check the SNI between handshake packet and local vHost configuration. So it is the place where mismatch happen. Error Code 400, Bad Request will return to client.<https://wiki.apache.org/httpd/NameBasedSSLVHostsWithSNI>
 
Definitely, port number shouldn't be a part of SNI. Compare with Chrome do, we hope Python could handle the server_hostname precisely too. Calling self._get_hostport() again and setting the IP address to server_hostname without port number is our suggestion.
msg259364 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-02-02 06:36
I still cannot reproduce any problem. When using set_tunnel(), this is the request sent to the proxy:

b'CONNECT 128.6.42.21:8088 HTTP/1.0\r\n'
b'\r\n'

This is the SNI and request sent through the proxy to the end server:

>>> wrapped = context.wrap_socket(conn, server_side=True)
Requested server name '128.6.42.21'
>>> for line in wrapped.recv(3000).splitlines(keepends=True): print(repr(line))... 
b'GET /xx/ HTTP/1.1\r\n'
b'Host: 128.6.42.21:8088\r\n'
b'Accept-Encoding: identity\r\n'
b'\r\n'

Can you please provide some code that demonstrates your problem.
msg259365 - (view) Author: lirenke (lvhancy) Date: 2016-02-02 07:25
We use Python 2.7.9, and I have checked the httplib.py in 2.7.11 version same time. Notice that in set_tunnel(), self._tunnel_host have already updated by calling _get_hostport(), just like you said. It's different between 2.7.9 and 2.7.11.

Then, I replace the file and retry, the problem is gone. Maybe this is an fixed bug before which I don't notice.

I think it's time to upgrade python for us...
Thank you for your remind and help, Martin.
History
Date User Action Args
2016-02-02 07:30:48lvhancysetresolution: fixed
2016-02-02 07:25:50lvhancysetstatus: open -> closed

messages: + msg259365
2016-02-02 06:36:15martin.pantersetmessages: + msg259364
2016-02-02 05:41:34lvhancysetmessages: + msg259361
2016-02-01 02:28:39martin.pantersetnosy: + martin.panter

messages: + msg259307
stage: test needed
2016-01-29 12:35:46lvhancycreate