With TLS it is possible to have the client use an extension (defined in
RFC 4366, and RFC 3546 before that) to indicate to the server which
hostname it believes it is talking to.  The server can then choose TLS
certificates accordingly.  This makes virtual-hosting possible.  Most
modern GUI web-browsers support making use of this extension, Server
Name Indication (SNI).

OpenSSL 0.9.8f onwards have optional support for this; OpenSSL needs to
have been built with "enable-tlsext" in EXTRACONFIGURE.  If that is not
present, then there's a guard macro defined to say it's absent.

This patch, against Python 2.6.1, adds to the standard ssl module the
ability to set the extension, using server_hostname as a arg in relevant
places.  This is only set for client connections and will silently be
ignored if the OpenSSL library does not support it.

I have tested this on FreeBSD 7.0/amd64 with OpenSSL 0.9.8k when talking
to Apache 2.2.x with the SNI patches from https://sni.velox.ch/.  Below
is my simple test program, to dump raw HTTP results back.  With this, I
can connect to various local https vhosts and get the correct content back.

I am not a Python core dev and not too enthusiastic at the thought of
grabbing latest svn to port this across; I hope that it's still of use.

=============
import socket
import ssl
import sys

def dump_https_page(hostname, uri='/'):

  sock = socket.socket(socket.AF_INET)
  s = ssl.SSLSocket(sock=sock,
                    ca_certs='/etc/ssl/certs',
                    server_hostname=hostname)
  print 'have socket'
  s.connect((hostname, 443))
  print 'connected'

  print >>s, 'GET %s HTTP/1.0\r\nHost: %s\r\nConnection: close\r\n\r\n' % (
      uri, hostname),

  t = s.read()
  while t:
    print t,
    t = s.read()

if __name__ == '__main__':
  for x in sys.argv[1:]:
    dump_https_page(hostname=x)

Note: this previous work is client-side only, as noted in the body of
the report.  I'll look into what's needed for clean server-side support too.

patch against TRUNK (2.7) with self tests and doco. Essentially the same
code as pdp with a SSLv2 check before using the SNI extension.

Contains some spacing cleanups that where highlighted by vim.

py3k version

current self tests cannot fully test the existence of the SNI extension
as there is no server side support.

This client script run with argument sni.velox.ch will show the "Great!
Your client ....its ClientHello: sni.velox.ch" on the output.

The small deficiency with these patches is that the specified
server_hostname is almost always the hostname that is used in the socket
pair of connect. Is it appropriate to grab the hostname value and use it
in the SNI extension header?

(Sorry for dropping this, lost available time)

I see your point.  OTOH, use of SNI needs to be something that can be 
disabled and people need to be able to connect to host A while supplying 
host B, not necessarily using IP addresses for the specificity.  Use-
case example: someone has a service "www" hosted on ["www-1", "www-2", 
"www-3"].  They have an SSL certificate for "www" and they want to have 
a health-checker which probes for "working service, all certs valid and 
not about to expire".

Unless s.connect() gains a keep_original_hostname=False option (?), this 
is hard to do.

Then there's the principle of least surprise -- while it would be nice 
to get SNI working automatically for everyone, it's still plausible that 
amongst the various TLS servers out there are some which break horribly 
for SNI and upgrading Python shouldn't break the tools in use.

So I tend towards favouring "make use of the newer, less well tested, 
protocol feature something that has to be explicitly enabled", even if 
it adds a line to boilerplate -- if SNI ever takes over the world (as 
Host: headers in HTTP have) then it can be defaulted on in future 
perhaps?

In which case, if the default is to change, the API should be sorted 
now, so perhaps connect() should take an override_server_hostname=False 
flag, which will make it pass the connect() hostname parameter instead 
of self.server_hostname in the call to _ssl.sslwrap()?  With checking 
for an IP address?

Hey Phil,
> (Sorry for dropping this, lost available time)
know the feeling :-(

> use of SNI needs to be something that can be disabled
maybe. See small rational below:

> and people need to be able to connect to host A while supplying host B
This seems to be a fringe case for usage and I seem to thing adding this
would overly complicate the API. When testing hosts you would have all
the names in DNS I'd assume.

> Unless s.connect() gains a keep_original_hostname=False option (?),
this is hard to do.
Is this starting to look too complicated?

Options for client side SNI:
1. wrapssl() - defaults to SNI enabled on SSL2/TLS1 connections (compile
time/module level/object variable disable if really needed?)
2. wrapssl(server_hostname=True/False) - enable SNI using the connect
hostname (if domainname and not IP/socket)
3. wrapssl(server_hostname=True/False/String) - True - enable SNI as
above, False/None- don't use SNI or use hostname if a String.
4. wrapssl(server-hostname=String)
5. connect() should an override_server_hostname=False
more?

> Then there's the principle of least surprise -- while it would be nice
to get SNI working automatically for everyone, it's still plausible that
amongst the various TLS servers out there are some which break horribly
for SNI and upgrading Python shouldn't break the tools in use.

Small rational to enable SNI by default:
1. SNI probably won't break too much stuff.
It was only in July 2009 that Apache-2.2.12 got officially released with
proper SNI support. Patches existed before then however deployment was
limited. mod_gnutls did implemented this earlier however I never thought
this was widely used.
Vista IE7 got SNI support in ~2005
(http://blogs.msdn.com/ie/archive/2005/10/22/483795.aspx)
FF got SNI support in the 2.0 release (October 24, 2006, wikipedia) 
(https://bugzilla.mozilla.org/show_bug.cgi?id=116169#c26)
The OpenSSL server side API for SNI required the registration of a
callback to receive the SNI name. It is possible that there are some
faulty implementations in this part of the server code. The default case
that a TLS server doesn't account for SNI is very safe as it won't ever
get a callback for it.
The existence of wide spread client side support with limited server
support hasn't broken the web.

> So I tend towards favouring "make use of the newer, less well tested, 
protocol feature something that has to be explicitly enabled", even if 
it adds a line to boilerplate -- if SNI ever takes over the world (as 
Host: headers in HTTP have) then it can be defaulted on in future 
perhaps?

2. I think with p3k there is an opportunity to put something in that may
break stuff. The release noted didn't guarantee stuff would work compatibly.

3. supporting SNI clients by default may actually break less stuff that
not supporting SNI client. e.g. Webhosting companies may embrace the SNI
features of Apache for maximum IP utilization breaking the non-SNI aware
clients (which won't be the majority of web browsers).

> With checking for an IP address?
To be RFC compliant IP addresses shouldn't be used in SNI. Apart from
socket family checking (AF_INET/AF_INET6) and doing a regex on the name
I couldn't see an easy way to do this even looking at the low level
socketmodule.c file. Maybe I need to look deeper. I could cheat and look
at the Firefox crypto (NSS) code though.

just my current thoughts

wrapssl(server_hostname=True/False/String) looks good to me.

Your arguments for enabling by default are compelling, for P3k.

Too late for 2.7 now, but looks like a good idea.

The patch probably needs refreshing now that first SSL contexts are in.

I wonder whether a combined boolean/string flag is really the best solution.

I think we could instead enable SNI by default and add an optional "server_hostname" to set the hostname to SSLContext.wrap_socket(), so that people can explicitly set the hostname; and otherwise take it, if possible, from the argument given to connect().

We can also add an "enable_sni" attribute to SSLContext (True by default) to allow selective disabling. This attribute would raise an exception if SNI support isn't available, which would be a way to test for it.

Here's another possible approach:

ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
ctx.set_tlsext_host_name("foo.bar")
skt = ctx.wrap_socket(socket.socket())
skt.connect("bar.baz")

This makes it obvious what the SNI hostname is and what the TCP address to connect to is, and they can easily be different.

> msg106323 - Author: Antoine Pitrou (pitrou) Date: 2010-05-22 20:17

I quite like your proposed alternative here. Not sure when/if I'll get to implement this.

> msg106324  - Author: Jean-Paul Calderone (exarkun) Date: 2010-05-22 22:17
Sorry I don't like this as much. I believe following the RFC for TLS SNI should be implicit and not something the programmer need to put effort into achieving. I acknowledge this approach does go against some explicit behaviour programming quality metrics.

> ctx = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
> ctx.set_tlsext_host_name("foo.bar")

Well, the hostname should be specific to a connection, so I'm not sure it makes sense to set it on the context.
(besides, the OpenSSL APIs only allow it to be set on the SSL structure)

> Sorry I don't like this as much. I believe following the RFC for TLS SNI should be implicit and not something the programmer need to put effort into achieving. I acknowledge this approach does go against some explicit behaviour programming quality metrics.

It's almost always wrong for Python to enforce a particular *policy*, particularly in a very low level API (which is what the ssl module should be).  Python's main job is to make it *possible* to do things.  It's the application developer's job to decide what things should be done.

It would be entirely appropriate, though, for a higher-level interface (for example, the httplib module) to take care of this itself and not require users to explicitly specify things separately.

> Well, the hostname should be specific to a connection, so I'm not sure it makes sense to set it on the context.

That doesn't make sense to me.  For example, consider the case where you're talking to a web service.  The hostname lookup might result in 10 A records, which you then drop into a connection pool.  Your application doesn't care which server you talk to (and maybe it talks to serveral, or all, of them).  But it does want to specify the same hostname for each.

> (besides, the OpenSSL APIs only allow it to be set on the SSL structure)

Nope, I checked before making the suggestion.  There's an SSL_CTX_ version of this API (in addition to the SSL_ version).

> Nope, I checked before making the suggestion.  There's an SSL_CTX_ version of this API (in addition to the SSL_ version).

Sorry, I just checked again, and it seems you're right.  Perhaps I saw SSL_CTX_set_tlsext_servername_callback and got the two confused.

> Python's main job is to make it *possible* to do things.  It's the
> application developer's job to decide what things should be done.

> It would be entirely appropriate, though, for a higher-level interface 
> (for example, the httplib module) to take care of this itself and not 
> require users to explicitly specify things separately.

Ok, I find this argument rather convincing. Also, enabling implicit SNI with the connect() argument could make user code stop working if he decides to pass the IP instead, without him being able to diagnose precisly what happens.

As you said, httplib/urllib should probably enable client-side SNI by default.

Here is a patch for py3k, including http.client and urllib support.

Committed with docs in r85793.

I see the patch has been applied python3 in r85793, but is there any chance there will also be patches for python 2.6 or 2.7? And if so, what release of python (any version) might this patch be included in?

> I see the patch has been applied python3 in r85793, but is there any
> chance there will also be patches for python 2.6 or 2.7

No, Python 2 only receives bug fixes.

And python3? Any idea which version the patch will be included there?
This might be a good reason to finally take action on migrating my code from
python 2.7 to python 3.

On 11 August 2011 18:49, Antoine Pitrou <report@bugs.python.org> wrote:

>
> Antoine Pitrou <pitrou@free.fr> added the comment:
>
> > I see the patch has been applied python3 in r85793, but is there any
> > chance there will also be patches for python 2.6 or 2.7
>
> No, Python 2 only receives bug fixes.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5639>
> _______________________________________
>

> And python3? Any idea which version the patch will be included there?

It was included in Python 3.2.