SSL certificate verification fails for "www.verisign.com" when using the cert list from Firefox. Other sites ("google.com", "python.org") verify fine.

This may be related to a known, and fixed, OpenSSL bug. See:

http://rt.openssl.org/Ticket/Display.html?id=2732&user=guest&pass=guest
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640

Some versions of OpenSSL are known to be broken for cases where there multiple valid certificate trees. This happens when one root cert is being phased out in favor of another, and cross-signing is involved.

Python ships with its own copy of OpenSSL on Windows. Tests
for "www.verisign.com"

Win7, x64:

Python 2.7.9 with OpenSSL 1.0.1j 15 Oct 2014. FAIL
Python 3.4.2 with OpenSSL 1.0.1i 6 Aug 2014. FAIL
openssl s_client -OpenSSL 1.0.1h 5 Jun 2014 FAIL

Ubuntu 14.04 LTS, x64, using distro's versions of Python:

Python 2.7.6 - test won't run, needs create_default_context
Python 3.4.0 with OpenSSL 1.0.1f 6 Jan 2014. FAIL
openssl s_client OpenSSL 1.0.1f 6 Jan 2014 PASS

That's with the same cert file in all cases. The OpenSSL version for Python programs comes from ssl.OPENSSL_VERSION.

The Linux situation has me puzzled. On Linux, Python is supposedly using the system version of OpenSSL. The versions match. Why do Python and the OpenSSL command line client disagree? Different options passed to OpenSSL by Python?

A simple test program and cert file are attached. Please try this in your environment.

Add cert file for testing. Source of this file is

http://curl.haxx.se/ca/cacert.pem

To try this with the OpenSSL command line client, use this shell command:

openssl s_client -connect www.verisign.com:443 -CAfile cacert.pem

This provides more detailed error messages than Python provides.

"verify error:num=20:unable to get local issuer certificate" is the OpenSSL error for "www.verisign.com". The corresponding Python error is "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)."

I have this problem too.

Debian jessie/sid
Python 2.7.8 (default, Nov 18 2014, 14:57:17)
Python 3.4.2 (default, Nov 13 2014, 07:01:52)

This may be related to a known, and fixed, OpenSSL bug.

Where do you see that the bug is fixed?

In https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640
it says :

FIX:
Fixed in Ubuntu 14.04 apparently.
Openssl upstream, see http://rt.openssl.org/Ticket/Display.html?id=2732

But I think the person who wrote that launchpad note was mistaken, as
the rt.openssl.org ticket still is marked open when I looked at it.

The "fix" in Ubuntu was to the Ubuntu certificate store, which is a directory tree with one cert per file, with lots of symbolic links with names based on hashes to express dependencies. Python's SSL isn't using that. Python is taking in one big text file of SSL certs, with no link structure, and feeding it to OpenSSL.

This is an option at

 SSLContext.load_verify_locations(cafile=None, capath=None, cadata=None)

I've been testing with "cafile". "capath" is a path to a set of preprocessed certs laid out like the Ubuntu certificate store. It may be that the directory parameter works but the single-file parameter does not. It's possible to create such a directory from a single .pem file by splitting the big file into smaller files (the suggested tool is an "awk" script) and then running "c_rehash", which comes with OpenSSL. See "https://www.openssl.org/docs/apps/c_rehash.html"

So I tried a workaround, using Python 3.4.0 and Ubuntu 14.04 LTS. I broke up "cacert.pem" into one file per cert with the suggested "awk" script, and used "c_rehash" to build all the links, creating a directory suitable for "capath". It didn't help. Fails for "verisign.com", works for "python.org" and "google.com", just like the original single-file test. The "capath" version did exactly the same thing as the "cafile" version.

Python is definitely reading the cert file or directories; if I try an empty cert file or dir, everything fails, like it should.

Tried the same thing on Win7 x64. Same result. Tried the command line openssl tool using the cert directory. Same results as with the single file on both platforms.

So that's not it.

A fix to OpenSSL was proposed in 2012, but no action was taken:

http://rt.openssl.org/Ticket/Display.html?id=2732 at
"Wed Jun 13 17:15:04 2012 Arne Becker - Correspondence added".

Any ideas?

Python's SSL isn't using that. Python is taking in one big text file
of SSL certs, with no link structure, and feeding it to OpenSSL.

Python's SSL is not "taking" anything:

>>> r = urlopen('https://www.verisign.com')
>>> r.read(10)
b' <!DOCTYPE'

It's only if you feed it that particular CA file that you get the issue:

>>> cafile = 'cacert.pem'
>>> r = urlopen('https://www.verisign.com', cafile=cafile)
[...]
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>

You can *also* feed it a CA directory by using the "CApath" argument (not "CAfile").

Now it remains to be seen why "openssl s_client" works with the file nevertheless.

John, neither Python nor OpenSSL are shipped with certificates.

Python uses certificates from operating system. We decided against our own certificate store because we wanted to avoid exactly this kind of trouble. If Python can't verify a certificate then you have to update the certificate storage of your OS.

On Linux and BSD Python, curl, wget and most other system tools use the OS's cert store. On Windows Python uses the same store as the IE, Chrome and other apps. Contrary to IE Python doesn't enforce cert store updates.

You can reproduce the problem with curl, too. The first call uses the OS' store, the second overwrite the default store.

$ curl https://www.verisign.com

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp curl https://www.verisign.com

So requests is running into this issue as well (see: https://github.com/kennethreitz/requests/issues/2455, https://github.com/kennethreitz/requests/issues/2456). With the specific code in Cory Benfield's comment (see: https://github.com/kennethreitz/requests/issues/2455#issuecomment-75773677) and the certificate file that requests 2.5.2 used (see: https://github.com/kennethreitz/requests/blob/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem, raw: https://raw.githubusercontent.com/kennethreitz/requests/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem) we can reproduce a similar problem on all versions of Python.

At the moment, we're investigating the possibility that it has to do with cross-signed certificates (see: http://openssl.6102.n7.nabble.com/Problems-with-cross-signed-certificates-and-Authority-Key-Info-td52280.html). We have a number of servers that we can reproduce this against and it is not reproducible using openssl s_client which means it is an issue with how Python has written its openssl compatibility layer.

Ok, this is really a bug in the cert bundle provided by requests and Firefox.

With requests 2.5.1:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> ok

With requests 2.5.2:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> Verify return code: 20 (unable to get local issuer certificate)

and it is not reproducible using openssl s_client

I have determined that s_client is buggy. It will always load the system certs *if and only if* you also pass it a valid custom CA cert (which is the reverse of what's expected).

This is where it happens (in apps/s_client.c):

if ((!SSL_CTX_load_verify_locations(ctx, CAfile, CApath)) ||
    (!SSL_CTX_set_default_verify_paths(ctx))) {
    /*
     * BIO_printf(bio_err,"error setting default verify locations\n");
     */
    ERR_print_errors(bio_err);
    /* goto end; */
}

This is why I forced SSL_CERT_* to empty locations in the examples above, so that only the custom CA bundle is used.

It appears it's not actually an issue with the CA Bundle, but I don't think it's actually an issue with Python, though Python might be in the best situation to try and fix it...

Basically, it appears that OpenSSL does not look inside the trust root for any certificate served by the server. In this case the sites have a chain that looks like A -> B -> NEW ROOT being served by the server, and NEW ROOT is also signed by OLD ROOT. If I construct the chain being sent from the server so it doens't have NEW ROOT, then everything works, but if the chain being sent from the server has NEW ROOT, then OpenSSL will only trust it if OLD ROOT is in the trust bundle. In this case Mozilla (and requests) has NEW ROOT in the trust bundle but not OLD ROOT, becuase OLD ROOT is a 1024 bit key.

Antione closed this, as a not python error, as
if you do not pass a valid certificate to openssl s_client
it will not read the system certificates, which is clearly
utterly surprising and nuts.

The problem, as I see it, is that fixing this clear
absurdity may not fix a different underlying problem. So this
one may need reopening when the real error us revealed.
See if John Nagel's code works ...

The problem specifically is that OpenSSL only uses a *root* in the trust store as an anchor. That means any certificate that is signed by another certificate will not terminate the chain of trust. Browsers do better here, by trusting the entirety of the trust store, regardless of whether or not it's a root certificate.

Donald is correct: this is not really Python's fault, it's OpenSSL's.

I've reported this as an update to OpenSSL bug bpo-2634.
(http://rt.openssl.org/Ticket/Display.html?id=2634)
Now we have to follow up there.

This bug should probably be set to "pending", not "closed". The problem is upstream, but OpenSSL is the Python libraries' choice, not the users'.
Python for Windows ships with its own copy of OpenSSL, so when (if) OpenSSL is fixed, the Python Windows distros will need an update.

So it seems like https://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3621 includes a fix that we may be able to update Python to use (safely) by default. If we don't then this will continue to be an issue.

Other references:

• https://bugzilla.redhat.com/show_bug.cgi?id=1166614

For now RedHat is keeping the 1024-bit certificates around for backwards compatibility and only because that option isn't set by default.

There actually *is* an API that can be set that will cause OpenSSL to use the shortest trust path it can, however it's only available in OpenSSL 1.0.2+ which means it'll solve it for a handful of people but not the bulk of people.

I'm attaching a patch that does what Donald suggests.

With the patch the flag is always set. Are there any possible side effects? IMHO it's better to add a store_flags property and make the feature optional.

It looks like the existing verify_flags param is actually the same thing, so we can just use it. That said, I think this should be on by default, I can't think of a scenario you don't want it.

My reading of the OpenSSL issue is that there are no negative side effects from turning this on.

New changeset 7f64437a707f by Benjamin Peterson in branch '3.4':
enable X509_V_FLAG_TRUSTED_FIRST when possible (closes bpo-23476)
https://hg.python.org/cpython/rev/7f64437a707f

New changeset 37da00170836 by Benjamin Peterson in branch '2.7':
enable X509_V_FLAG_TRUSTED_FIRST when possible (closes bpo-23476)
https://hg.python.org/cpython/rev/37da00170836

New changeset 442e2c357979 by Benjamin Peterson in branch 'default':
merge 3.4 (bpo-23476)
https://hg.python.org/cpython/rev/442e2c357979

Benjamin, can you please add at least a comment describing why you added the flag? We have enough obscure-looking code in _ssl.c as it is.

Uh, the comment is already there. I don't know how I missed that. Sorry.

The Windows binaries of Python 2.7.9 are compiled with OpenSSL 1.0.1j. The feature is only available in OpenSSL > 1.0.2. The next version of Python must be compiled with 1.0.2 or better. Otherwise the bug pops up again.

bpo-23593 opened to request Windows and OS X installer OpenSSL updates to 1.0.2

Will this be applied to the Python 2.7.9 library as well?

It was merged to the 2.7 branch, so it'll be released as part of 2.7.10.