classification
Title: ssl.wrap_socket (cert_reqs=...), getpeercert, and unvalidated certificates
Type: enhancement Stage: resolved
Components: Library (Lib), SSL Versions: Python 3.7
process
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, jcea, mpb, pitrou, r.david.murray, sascha_silbe, underrun
Priority: normal Keywords:

Created on 2013-06-24 19:04 by mpb, last changed 2018-02-25 20:24 by christian.heimes. This issue is now closed.

Messages (12)
msg191796 - (view) Author: mpb (mpb) Date: 2013-06-24 19:04
At present (Python 2.7.[45] and 3.3.[12]), the cert_reqs parameter of ssl.wrap_socket can be one of:

ssl.CERT_NONE
ssl.CERT_OPTIONAL
ssl.CERT_REQUIRED

I would find the following additional modes to be useful:
ssl.CERT_OPTIONAL_NO_VERIFY
ssl.CERT_REQUIRED_NO_VERIFY

In these cases, the server's certificate would be available via the .getpeercert () method, even if the certificate does not pass verification.

The use case for these modes would be connecting to servers, some of which may use valid certificates, and others of which may be using self signed certificates.

Another use case might be an "ssh-like" mode of operation.  ssh will warn the user of possible man-in-the-middle attacks if a server's public key has changed.

Thanks!
msg191797 - (view) Author: mpb (mpb) Date: 2013-06-24 19:06
(Oops, I changed the title when I meant to do a search.  Changing it back now.)
msg191823 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-24 23:30
I'm setting the version to 3.4 as this is a feature request. 2.7 and 3.3 are in feature freeze mode.

OpenSSL doesn't support our idea out of the box. OpenSSL either verifies the peer's certificate and chain or doesn't verify the peer's certificate and chain. Optional and required verification makes only a different for client side certs. Server side certs are always verified in both modes. See http://www.openssl.org/docs/ssl/SSL_CTX_set_verify.html#NOTES

When you are talking to a server the peer's certificate is always available, even in SSL_VERIFY_NONE mode. The server cert's public key is required to asymmetrically encrypt part of the session key. It's Python's ssl module that doesn't return the cert information in getpeercert() when SSL_CTX_get_verify_mode() doesn't have SSL_VERIFY_PEER. You can still get the DER encoded peer cert with getpeercert(True).

Now for something completely different: Without verification and the correct root cert it's not possible to get the root cert of a peer's chain (see issue #18233). AFAIK SSL doesn't provide the full root cert as part of the peer chain because the other side is suppose the have a copy of the chain root anyway. Different story, though.
msg191830 - (view) Author: mpb (mpb) Date: 2013-06-25 03:13
Hi Christian, thanks for the prompt response.

Sorry about choosing the wrong versions - I wasn't thinking that enhancements should target future versions, but of course that makes sense.

After submitting the enhancement request, I did dig into the OpenSSL docs, and, as Christian points out, I discovered that OpenSSL is not designed in a way that makes it easy to implement the enhancement.

Aside: Interestingly, it looks easier to implement the enhancement in PolarSSL, and probably also in MatrixSSL and CyaSSL.  Of course, that's not really an option.  I did not look at GnuTLS.

Thanks for the pointer about being able to get the server's DER certificate.  That will be useful.  Is there some reason to return DER but not PEM?  Or is this perhaps a bug that could be fixed in a future version of Python's ssl module?

Christian wrote: "Optional and required verification makes only a differen[ce] for client side certs."

I believe there is one small exception:  With SSL_VERIFY_NONE, a client will continue talking with a server with an invalid certificate.  With SSL_VERIFY_PEER, when a client fails to verify the server's certificate, the client will terminate the connection.

Ideally, I would like a client to be able to get both of  the following from the API: (a) the server's certificate (and chain?), and (b) whether or not the certificate (and chain?) is valid (against a given sets of root certs).

Similarly, I would like a Python server to be able to get both of: (a) the client's certificate, and (b) whether the certificate is valid (against a given set of root certs).

In the latter case, it seems that OpenSSL is even more restrictive!  With SSL_VERIFY_NONE, the server will not request (and presumably therefore not even receive) a certificate.  With SSL_VERIFY_PEER, the server will terminate the connection if the client's certificate does not validate!  Very inconvenient!

Interestingly, I believe I have worked around this limitation in OpenSSL using M2Crypto (which is built on top of OpenSSL), by installing my own verifier that overrides the built-in verifier.  This is done as follows:

    import M2Crypto.SSL
    ctx  = M2Crypto.SSL.Context ()
    ctx.load_cert ('var/cert.pem')
    def verify (*args):  return True
    ctx.set_verify (M2Crypto.SSL.verify_peer, 10, verify)

After doing this, both the client and the server can see each other's certificates, even if those certificates are invalid.  (Of course I'll have to write my own verifier.  "return True" is only useful for testing purposes.)

I'm not sure how much of this functionality the Python developers might be interested in putting into Python 3.4?  Given that M2Crypto does not work with Python 3.x at all (at least not yet?), I am interested in finding something that will work with Python 3.x and give me the functionality I want.

I can probably help with the C OpenSSL code, if needed.  However, I have no experience writing Python bindings.

Your thoughts?  Thanks!
msg191831 - (view) Author: mpb (mpb) Date: 2013-06-25 03:33
Oh, I see.  getpeercert (binary_form) is not DER vs. PEM, it is DER vs. dict.
msg191857 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-25 13:00
> Thanks for the pointer about being able to get the server's DER certificate.  That will be useful.  Is there some reason to return DER but not PEM?  Or is this perhaps a bug that could be fixed in a future version of Python's ssl module?

It doesn't matter how getpeercert() returns the raw cert. PEM and DER
encode the X.509 same in ASN.1 notation. PEM is just the base64
representation with some line breaks surrounded BEGIN/END CERTIFICATE lines.

> Christian wrote: "Optional and required verification makes only a differen[ce] for client side certs."
> 
> I believe there is one small exception:  With SSL_VERIFY_NONE, a client will continue talking with a server with an invalid certificate.  With SSL_VERIFY_PEER, when a client fails to verify the server's certificate, the client will terminate the connection.

You are referring to server side certs, that is the server provides a
cert and the client verifies it. I was talking about client side cert
(aka server mode), a rarely used mode in which the client side provides
a cert to the server. In CERT_OPTIONAL mode the client is allowed to
send no cert. CERT_REQUIRED is mapped to SSL_VERIFY_PEER |
SSL_VERIFY_FAIL_IF_NO_PEER_CERT.

> Ideally, I would like a client to be able to get both of  the following from the API: (a) the server's certificate (and chain?), and (b) whether or not the certificate (and chain?) is valid (against a given sets of root certs).

In client mode you already have the server's cert and as much of the
chain as the peer (server) provides. SSL_VERIFY_NONE makes no difference
because the server cert is used in the ClientHello / ServerHello
handshake and session key negotiation.

Perhaps you are a bit confused by Python's ssl API. Somebody once made
the choice to return an empty dict from getpeercert(False) when
SSL_VERIFY_NONE is used. That's all. :)
http://hg.python.org/cpython/file/76196691b5d0/Modules/_ssl.c#l1098

> I'm not sure how much of this functionality the Python developers might be interested in putting into Python 3.4?  Given that M2Crypto does not work with Python 3.x at all (at least not yet?), I am interested in finding something that will work with Python 3.x and give me the functionality I want.

I'd rather not implement a full wrapper for X509_STORE_CTX and X509
certs. It's way too much code, super complex and easily confuses even
experienced developers. Python's ssl module is limited to core
functionality by design and choice.

However I might be intrigue to implement support for
SSL_CTX_set_cert_verify_callback() or SSL_CTX_set_verify().
SSL_CTX_set_cert_verify_callback() has more potential, e.g.

def cert_verify_callback(sslsocket, storectx, verify_ok):
    context = sslsocket.context

storectx is a minimal X509_STORE_CTX object and verify_ok the boolean
return value of X509_verify_cert(). Without a cert verify callback
OpenSSL just uses the return value of X509_verify_cert()
(ssl/ssl_cert.c:ssl_verify_cert_chain()). sslsocket gives you access to
the peer's cert and chain (with #18233).

The callback could be used for multiple uses cases like cert
blacklisting, too.
msg191882 - (view) Author: mpb (mpb) Date: 2013-06-25 20:08
Christian wrote:
> sslsocket gives you access to the peer's cert and chain (with 
> #18233).

Very interesting (and useful).  I've mostly been working with Python
2.7, and I had not fully noticed that Python 3.2+ has a ssl.SSLContext
class.

> I'd rather not implement a full wrapper for X509_STORE_CTX and X509 
> certs. It's way too much code, super complex and easily confuses even 
> experienced developers. Python's ssl module is limited to core 
> functionality by design and choice.

> However I might be intrigue to implement support for
> SSL_CTX_set_cert_verify_callback() or SSL_CTX_set_verify().

SSL_CTX_set_verify() seems (mostly) redundant SSLContext.verify_mode.  
Or am I missing something?

> SSL_CTX_set_cert_verify_callback() has more potential, e.g.
> 
> def cert_verify_callback(sslsocket, storectx, verify_ok):
>     context = sslsocket.context
>
> storectx is a minimal X509_STORE_CTX object and verify_ok the boolean
> return value of X509_verify_cert(). Without a cert verify callback
> OpenSSL just uses the return value of X509_verify_cert()
> (ssl/ssl_cert.c:ssl_verify_cert_chain()).

I believe support for SSL_CTX_set_cert_verify_callback() would allow
customized certificate verification, which is what I am looking for.
msg193569 - (view) Author: Derek Wilson (underrun) Date: 2013-07-22 22:21
Custom cert validation may make sense in edge cases, so this looks interesting.

But I got here looking to file a bug on the returning empty dict from SSLContext.getpeercert - I don't feel like that makes sense. Its not like a peer cert doesn't exist just because it doesn't pass verification. 

And I know I can get the binary DER cert, but then I need to figure out how to parse it myself? pyasn1 makes me cry. 

It would be really nice if we could get the decoded-into-dict peer cert even when verification fails or when CERT_NONE is set. If it isn't possible (or advisable) for getpeercert to return the dict, exposing cert decoder would be really useful.

Is this related enough to this request or should I file a separate issue?
msg193603 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-23 14:00
getpeercert() has a crappy API to begin with, but we can't change its behaviour for fear of breaking existing code (and, even, breaking it security-wise). Adding a parameter would make the API even more awful.

Which is why I support Christian's idea of exposing a new API, either:
- to expose the full cert chain (even if not validated)
- or to set the cert verify callback
msg203181 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-17 14:42
I may address the issue in my PEP for Python 3.5. Python 3.4 beta 1 will be released next week and no new features are allowed in beta and RC phase.
msg262757 - (view) Author: Sascha Silbe (sascha_silbe) Date: 2016-04-01 19:15
Has there been any progress on this? For my application I'd very much like "ssh-like" operation, using the public key itself as identifier rather than requiring some kind of automated CA setup.

Being able to set a custom verification callback would be great, but just being able to cause a dummy callback that accepts any certificate to be used would go a long way. The validation could be done after the connection was established in this case. For some applications, that may even be the best approach, presenting any verification error via the application layer (e.g. HTTP) where they are closer to the problem domain and thus make more sense to the user.
msg312852 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-25 20:24
BPO #31372 and #18369 provide the necessary bits and pieces for your request. I didn't have enough time to finish both in time for 3.7 feature freeze. Hostname verification improvements and TLS 1.3 were more important.

I'm closing this issue because ssl.wrap_socket() is deprecated and will be removed in the future.
History
Date User Action Args
2018-02-25 20:24:29christian.heimessetstatus: open -> closed
resolution: rejected
messages: + msg312852

stage: resolved
2016-09-15 08:11:02christian.heimessetassignee: christian.heimes
components: + SSL
versions: + Python 3.7, - Python 3.5
2016-06-12 11:22:46christian.heimessetassignee: christian.heimes -> (no value)
2016-04-01 19:15:06sascha_silbesetnosy: + sascha_silbe
messages: + msg262757
2013-11-20 16:48:02christian.heimessetassignee: christian.heimes
2013-11-17 14:42:04christian.heimessetmessages: + msg203181
versions: + Python 3.5, - Python 3.4
2013-08-14 19:05:29jceasetnosy: + jcea
2013-07-23 14:00:35pitrousetnosy: + pitrou
messages: + msg193603
2013-07-22 22:21:34underrunsetnosy: + underrun
messages: + msg193569
2013-06-25 20:08:54mpbsetmessages: + msg191882
2013-06-25 13:14:17r.david.murraysetnosy: + r.david.murray
2013-06-25 13:00:35christian.heimessetmessages: + msg191857
2013-06-25 03:33:32mpbsetmessages: + msg191831
2013-06-25 03:13:19mpbsetmessages: + msg191830
2013-06-24 23:30:36christian.heimessetmessages: + msg191823
versions: + Python 3.4, - Python 2.7, Python 3.3
2013-06-24 19:17:42pitrousetnosy: + christian.heimes
2013-06-24 19:06:19mpbsetmessages: + msg191797
title: ssl sni -> ssl.wrap_socket (cert_reqs=...), getpeercert, and unvalidated certificates
2013-06-24 19:05:33mpbsettitle: ssl.wrap_socket (cert_reqs=...), getpeercert, and unvalidated certificates -> ssl sni
2013-06-24 19:04:53mpbcreate