classification
Title: SSLSocket.getpeercertchain()
Type: enhancement Stage: patch review
Components: SSL Versions: Python 3.9
process
Status: open Resolution:
Dependencies: 18147 Superseder:
Assigned To: christian.heimes Nosy List: Hiroaki.Kawai, asmodai, chaen, chet, chrisburr, christian.heimes, dsoprea, dstufft, jcea, joernheissler, kwatsen, maker, miki725, mmasztalerczuk, njs, pitrou, underrun, zwol
Priority: normal Keywords: patch

Created on 2013-06-16 20:39 by christian.heimes, last changed 2020-06-30 09:17 by chrisburr.

Files
File name Uploaded Description Edit
ssl_peerchertchain.patch christian.heimes, 2013-06-16 20:39 review
ssl_peerchertchain2.patch christian.heimes, 2013-06-17 18:07 review
ssl_peercertchain3.patch mmasztalerczuk, 2016-10-11 15:42 review
Pull Requests
URL Status Linked Edit
PR 17938 open chrisburr, 2020-01-10 15:56
Messages (31)
msg191287 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-16 20:39
The patch implements a method getpeercertchain() on a SSLSocket. It returns the peer's certificate chain from the leaf cert to the root cert if available. It wraps SSL_get_peer_cert_chain().

SSL_get_peer_cert_chain() doesn't have to pull any additional data from the peer. The information is already exchanged for cert validation.
msg191352 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-17 18:07
As expected it is much harder to get the full certification chain from OpenSSL than I initially expected. SSL_get_peer_cert_chain() doesn't return the root CA's certificate. The new patch introduces a validation mode and uses X509_verify_cert(*X509_STORE_CTX) + X509_STORE_CTX_get1_chain() to build a full chain.
msg193424 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-20 23:42
From Rietveld review:

---
http://bugs.python.org/review/18233/diff/8422/Modules/_ssl.c#newcode1203
Modules/_ssl.c:1203: chain = X509_STORE_CTX_get1_chain(store_ctx);
This isn't appropriate for this method. Specifically, you are asking for
the peer cert chain, which purposefully does not include root CA certs
that you trust. What you are giving here a complete validate chain from
a peer cert to a trusted root. This is a valuable piece of information,
but should be returned via another method (perhaps exposed in python as
get1chain in SSLContext). But this method should always return the
result of SSL_get_peer_cert_chain if a peer cert chain is available.
---

You are making a good point. I'm either going to split it up into two function or provide a way to look up a cert by issuer.
msg199434 - (view) Author: Dustin Oprea (dsoprea) * Date: 2013-10-11 01:28
I was about to submit a feature request to add exactly this. The [second] patch works like a charm. When are you going to land on a particular resolution so that it can get committed in?



Dustin
msg199443 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-11 06:30
The patch needs a test, a proper doc, and reviewing.
msg199446 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-10-11 09:06
Sorry for the incorrect answer. I just noticed there was a test in the patch!
Further looking at it, I notice the new function is returning a tuple. Wouldn't it be better to return a list here?
msg199447 - (view) Author: Dustin Oprea (dsoprea) * Date: 2013-10-11 09:09
My two-cents is to leave it a tuple (why not?).



Dustin
msg203187 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-17 14:54
I'd rather return a list or tuple of X509 objects but #18369 won't be ready for 3.4. Ideas?
msg203189 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-17 15:00
@Dustin

> My two-cents is to leave it a tuple (why not?).

Because tuples are more used for struct-like data. Here we are returning an unknown number of homogenous objects, which generally calls for a list.

@Christian

> I'd rather return a list or tuple of X509 objects but #18369 won't be ready for 3.4. Ideas?

Unless the feature is really important to have right now, I think it's ok to defer it until we have X509 objects.
msg203190 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-17 15:04
It's just nice to have for debugging and extended verification.
msg205734 - (view) Author: Derek Wilson (underrun) Date: 2013-12-09 19:35
I could really use this sooner than later... and sometimes having a full-featured (or even secure) interface is not what you want.

Consider zmap and masscan etc and ssl mapping (similar to what the EFF did a couple years back - https://www.eff.org/observatory - but with the full chain instead of just the cert). The desire here would be low level, low overhead, no validation on the fly: All you want is the cert chain.

There are plenty of research and security applications where a simple wrapper around OpenSSL that returns DER bytes would be desirable. Please reconsider this patch for inclusion in 3.4 ...
msg244312 - (view) Author: Jeroen Ruigrok van der Werven (asmodai) * (Python committer) Date: 2015-05-28 15:58
Given that cryptography.io is fast becoming the solution for dealing with X.509 certificates on Python, I would like to add my vote to add my vote for this feature. Right now, getting the full chain in DER is what I am missing to complete a task at work.
msg244313 - (view) Author: Dustin Oprea (dsoprea) * Date: 2015-05-28 16:03
Forget it. This project is dead.

Dustin
On May 28, 2015 11:58 AM, "Jeroen Ruigrok van der Werven" <
report@bugs.python.org> wrote:

>
> Jeroen Ruigrok van der Werven added the comment:
>
> Given that cryptography.io is fast becoming the solution for dealing with
> X.509 certificates on Python, I would like to add my vote to add my vote
> for this feature. Right now, getting the full chain in DER is what I am
> missing to complete a task at work.
>
> ----------
> nosy: +asmodai
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue18233>
> _______________________________________
>
msg244314 - (view) Author: Dustin Oprea (dsoprea) * Date: 2015-05-28 16:07
Disregard. I thought this was something else.
msg278494 - (view) Author: Mariusz Masztalerczuk (mmasztalerczuk) * Date: 2016-10-11 15:42
Hello :)

I'm not sure why patches created by christian.heimes is not merged to python, but because last patch was created in 2013, I've created a new version of this patch.

What do you think about it?
msg280348 - (view) Author: Mariusz Masztalerczuk (mmasztalerczuk) * Date: 2016-11-08 21:05
ping! :) 
Could someone look at my changes? :)
msg293570 - (view) Author: Chet Nichols III (chet) Date: 2017-05-12 20:48
Is this dead at this point? Just stumbled upon it, and I'm hopeful that maybe there's still a chance, since it's still `open`. :)
msg293577 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-05-12 21:45
The ticket is dead for a very good reason. Past me was not clever enough and didn't know about the difference between the cert chain sent by the peer and the actual trust chain. The peer's cert chain is not trustworthy and must *only* be used to build the actual trust chain. X.509 chain trust chain construction is a tricky business.

Although I thought that peer cert chain is a useful piece of information, it is also dangerous. It's simply not trustworthy. In virtually all cases you want to know the chain of certificates that leads from a local trust anchor to the end-entity cert. In most cases it just happens to be the same (excluding root CA). But that's not reliable.
msg293580 - (view) Author: Dustin Oprea (dsoprea) * Date: 2017-05-12 22:07
Thanks for expounding on this, Christian. Assuming your assertions are
correct, this makes perfect sense.

Can anyone listening close this?

On May 12, 2017 17:45, "Christian Heimes" <report@bugs.python.org> wrote:

Christian Heimes added the comment:

The ticket is dead for a very good reason. Past me was not clever enough
and didn't know about the difference between the cert chain sent by the
peer and the actual trust chain. The peer's cert chain is not trustworthy
and must *only* be used to build the actual trust chain. X.509 chain trust
chain construction is a tricky business.

Although I thought that peer cert chain is a useful piece of information,
it is also dangerous. It's simply not trustworthy. In virtually all cases
you want to know the chain of certificates that leads from a local trust
anchor to the end-entity cert. In most cases it just happens to be the same
(excluding root CA). But that's not reliable.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue18233>
_______________________________________
msg293590 - (view) Author: Chet Nichols III (chet) Date: 2017-05-12 22:52
Oh yeah, definitely not trustworthy at all. In my case, I am not processing the peer chain to actually verify trust, but I am still interested in inspecting the chain.

Dangerous or not, and regardless of what almost all people should *actually* be doing, SSL_get_peer_cert_chain exists for a reason, just like SSL_get_peer_certificate exists for a reason. If Python includes a standard SSL library, it should be transparent in the interface it offers, for the mere reason that the library becomes more powerful.

If the overall consensus is that the library should protect most people against common pitfalls and security mistakes, then I guess that's the route to continue on. However, I would be disappointed that we would be blacklisting  the exposure of underlying library features based on the mere belief that people don't understand them enough!
msg293778 - (view) Author: Jörn Heissler (joernheissler) * Date: 2017-05-16 18:53
Hi,
I'd like to see this feature too.

My use case is a monitoring script to check the life time of the server certificate, including the chain. I would prefer to have a wrapper around SSL_get_peer_cert_chain.
I understand that this is *not* a verified chain. That's okay.

openssl-1.1 added a new function SSL_get0_verified_chain which may be safer for most applications. Is there any real difference to X509_STORE_CTX_get1_chain?

If you're worried about people misusing these functions, add a warning in the docs and point them to "get_peer_verified_chain"?
msg301525 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-06 21:42
Yes, from an application perspective there is an import difference between X509_STORE_CTX_get1_chain() and SSL_get0_verified_chain(). X509_STORE_CTX is a temporary object. It is only available during the handshake and while the trust chain is built and verified. Once the chain is verified, it is no longer available.

SSL_get0_verified_chain() sounds like an actual good solution. Thanks for pointing it out.
msg324917 - (view) Author: chaen (chaen) Date: 2018-09-10 10:13
There is another very valid use case which is even described by an RFC: https://www.ietf.org/rfc/rfc3820.txt

And openssl supports this RFC. 

These proxy certificates are heavily used in the world of high energy physics computing, and having the get_peer_cert_chain exposed in python would allow to use standard tools. 

And any hope this would also be backported to the python 2.7 branch ?
msg357540 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2019-11-27 00:13
There's another important use case for this, that hasn't been discussed here. If you want to use openssl for TLS + the system trust store to verify certificates, then you need to disable openssl's certificate validation, perform the handshake, and then extract the certificate chain that there peer sent and pass it to the system native APIs to validate.

For this case, we don't need to do any validation or resolution on the chain – we just want to pull out the DER that the peer sent. AFAICT, the lack of this functionality is the one major blocker to using the system trust store with the 'ssl' module.
msg361076 - (view) Author: Kent Watsen (kwatsen) Date: 2020-01-30 17:26
I don't understand the concern issues being raised for this patch, and also may have a use-case not mentioned yet.

For the concern issue, as I understand it, the ability to call getpeercert() or the proposed getpeercertchain() is only after the TLS session has been established.  As such, the SSL socket already established that there exists a valid chain of trust.  Thus these methods are primarily to provide visibility into what the peer passed *after* it had been authenticated, right?

That said, the reason I want to access the entire certificate chain passed by the client (i.e., client cert auth) is in order to validate that the client's cert (which may include some intermediates) authenticates to a specific trust anchor, rather than the bag of trust anchors loaded into the SSLContext (via load_verify_locations()) in order to authenticate a multiplicity of clients, each potentially leading to a different trust anchor.  

Not having getpeercertchain() means that all no client cert may contain a chain, i.e., the clients only ever transmit the end-entity cert itself.  This is unfortunate because the underlying SSL socket actually allows clients to send chains, it's just the lack being able to access the peercertchain in my code that seems to limit the solution.
msg361085 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2020-01-31 00:37
> For the concern issue, as I understand it, the ability to call getpeercert() or the proposed getpeercertchain() is only after the TLS session has been established.  As such, the SSL socket already established that there exists a valid chain of trust.  Thus these methods are primarily to provide visibility into what the peer passed *after* it had been authenticated, right?

The issue is that "the cert chain provided by the client" and "the cert chain that was validated as trustworthy" might be different. For example, say you have trust roots A and B, and I have a leaf certificate with a valid signature by B. If I pass the chain [A, leaf cert], then openssl's trust validation code might look at that and say "yep, this leaf cert is signed by root B, I trust root B, cool, the connection is good. Also there's a random A cert here, but whatever, that doesn't change anything".

In this case, for your use case, you want to get back the chain [B, leaf cert], because that's the chain that was actually validated. If you instead got the chain the client gave you, then you might be fooled into thinking that this cert chained to root A, when it actually didn't, and make the wrong trust decision.

That's why we need to be careful to distinguish between these two possible chains.
msg361093 - (view) Author: Kent Watsen (kwatsen) Date: 2020-01-31 02:27
It seems that we're talking about the same thing, but I want the cert-chain the peer sent without any smarts, exactly how OpenSSL's SSL_get_peer_cert_chain() works and, importantly, without stapling any root chain certs the client did not send itself (though it's okay if the client did, in which case those certs should be included).

I'm not following your "I pass the chain [A, leaf cert]" comment, if leaf-cert is signed by B, then this should obviously fail.  Maybe you meant to say that A and B are loaded into a bag and that validation test is [bag, leaf-cert]?

Regardless, I don't think Python should coddle developers.  Assuming the docs are accurate, competent developers with crypto-clue will be fine.  Many crypto library docs encourage tourists to stay away.   That said, if smarts are wanted, let's choose a name that doesn't overlap with the existing OpenSSL name...get_authed_cert_chain() ?

But, please, can a "peer_cert_chain()" wrapping the OpenSSL call be release ASAP, buying time to ponder the merits of smart calls for another day?
msg361095 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2020-01-31 08:32
I'm not sure I agree about assuming that users will be able to work around these issues... I mean, nothing personal, I'm sure you're well-informed and maybe your code would work fine, but if you don't understand my example then how can you be entirely confident that you really understand all the risks that your code needs to watch out for?

Anyway, the proposed PR exposes both methods, so that's not a problem; it's just waiting on someone with openssl expertise like Christian to review it.
msg361120 - (view) Author: Kent Watsen (kwatsen) Date: 2020-01-31 15:23
I agree that having both would be best, but there is a world of difference between a must-have (peer_cert_chain) and what seems to be a nice-to-have (authed_peer_cert_chain).

My request for clarification was not that I don't understand bags, etc. (see my first message), but that I don't understand the concrete use case in mind.  That is, when is it that the app-logic would differ because the EE cert validated using one path versus another?

To explain the 'must-have' better, imagine one peer sending [A, B, C], where 'A' is the EE cert, and the other peer having TA [F, E, D], where 'F' is the self-signed root TA and 'D' is the Issuer that signed 'C'.  The complete chain is [A-F] and this is what the SSL-level code will use during the handshake.  But post-handshake, without peer_chain_cert(), there is NO WAY for the app-logic to create a valid chain.  This is broken, for the reason mentioned in my first message.
msg372627 - (view) Author: Zack Weinberg (zwol) * Date: 2020-06-29 20:42
I have yet another use case for the function implemented by this patch (i.e. retrieving the cert chain actually sent by the server, regardless of whether that gives a path to a trust anchor).  I'm implementing a network forensics tool, and one of the situations it's supposed to detect is when a man-in-the-middle is attempting to substitute its own cert for a site's "legitimate" cert (yes, possibly having suborned a public CA in order to do so).  To make all of the planned heuristics for this work correctly, I need to record exactly what came over the wire.

If it would be useful for me to dust off the patch and/or implement the _other_ function that people requested (retrieve the chain that OpenSSL concluded was a valid chain to an accepted trust anchor)  I can probably scare up time to do so in the next week or two.  I imagine it's too late for 3.8 patch releases at this point, but assuming I did this, could it make 3.9?
msg372673 - (view) Author: Chris Burr (chrisburr) * Date: 2020-06-30 09:17
Hi Zack, I've already opened a PR that is loosely based on this patch. If you have time to give it a review I'd appreciate the extra set of eyes.

https://github.com/python/cpython/pull/17938
History
Date User Action Args
2020-06-30 09:17:19chrisburrsetmessages: + msg372673
versions: + Python 3.9, - Python 3.8
2020-06-29 20:42:38zwolsetnosy: + zwol
messages: + msg372627
2020-01-31 15:23:17kwatsensetmessages: + msg361120
2020-01-31 08:32:38njssetmessages: + msg361095
2020-01-31 02:27:10kwatsensetmessages: + msg361093
2020-01-31 00:37:08njssetmessages: + msg361085
2020-01-30 17:26:52kwatsensetnosy: + kwatsen
messages: + msg361076
2020-01-13 17:10:31chrisburrsetnosy: + chrisburr
2020-01-10 15:56:01chrisburrsetpull_requests: + pull_request17346
2019-11-27 00:13:37njssetnosy: + njs
messages: + msg357540
2019-03-31 10:08:16Hiroaki.Kawaisetnosy: + Hiroaki.Kawai
2018-09-10 10:13:09chaensetnosy: + chaen
messages: + msg324917
2018-02-26 08:31:55christian.heimessetversions: + Python 3.8, - Python 3.7
2017-09-06 21:42:30christian.heimessetmessages: + msg301525
2017-05-16 18:53:15joernheisslersetnosy: + joernheissler
messages: + msg293778
2017-05-12 22:52:05chetsetmessages: + msg293590
2017-05-12 22:07:08dsopreasetmessages: + msg293580
2017-05-12 21:45:01christian.heimessetmessages: + msg293577
2017-05-12 20:48:23chetsetnosy: + chet
messages: + msg293570
2016-11-08 21:05:57mmasztalerczuksetmessages: + msg280348
2016-10-11 15:42:59mmasztalerczuksetfiles: + ssl_peercertchain3.patch
nosy: + mmasztalerczuk
messages: + msg278494

2016-09-15 07:48:34christian.heimessetassignee: christian.heimes
components: + SSL
2016-09-15 07:25:56christian.heimessetversions: - Python 3.6
2016-09-11 02:07:54miki725setnosy: + miki725
2016-09-08 15:42:23christian.heimessetversions: + Python 3.6, Python 3.7, - Python 3.4, Python 3.5
2016-06-12 11:23:15christian.heimessetassignee: christian.heimes -> (no value)
2015-05-28 16:07:07dsopreasetmessages: + msg244314
2015-05-28 16:03:38dsopreasetmessages: + msg244313
2015-05-28 15:58:45asmodaisetnosy: + asmodai
messages: + msg244312
2013-12-09 19:35:15underrunsetmessages: + msg205734
versions: + Python 3.4
2013-11-20 16:47:33christian.heimessetassignee: christian.heimes
2013-11-17 15:04:51christian.heimessetmessages: + msg203190
versions: + Python 3.5, - Python 3.4
2013-11-17 15:00:04pitrousetmessages: + msg203189
2013-11-17 14:54:20christian.heimessetmessages: + msg203187
2013-10-11 09:09:34dsopreasetmessages: + msg199447
2013-10-11 09:06:33pitrousetmessages: + msg199446
2013-10-11 06:30:31pitrousetnosy: + pitrou
messages: + msg199443
2013-10-11 01:28:42dsopreasetnosy: + dsoprea
messages: + msg199434
2013-08-24 22:37:35dstufftsetnosy: + dstufft
2013-07-24 15:26:34christian.heimeslinkissue18546 superseder
2013-07-20 23:42:20christian.heimessetmessages: + msg193424
2013-07-20 23:15:38underrunsetnosy: + underrun
2013-07-05 20:35:35makersetnosy: + maker
2013-06-20 02:26:38jceasetnosy: + jcea
2013-06-17 18:07:25christian.heimessetfiles: + ssl_peerchertchain2.patch

messages: + msg191352
2013-06-17 11:55:30pitrousetdependencies: + SSL: diagnostic functions to list loaded CA certs
2013-06-16 20:39:55christian.heimescreate