classification
Title: SSL cert verify fail for "www.verisign.com"
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Aaron.Meurer, Lukasa, alex, christian.heimes, demian.brecht, dstufft, giampaolo.rodola, icordasc, janssen, jcea, lac, nagle, ned.deily, pitrou, python-dev
Priority: normal Keywords: needs review, patch

Created on 2015-02-18 01:07 by nagle, last changed 2015-03-20 12:11 by jcea. This issue is now closed.

Files
File name Uploaded Description Edit
ssltest.py nagle, 2015-02-18 01:07 Test program to reproduce bug
cacert.pem nagle, 2015-02-18 01:08 Certificate file for testing.
store.diff alex, 2015-03-01 20:19 review
Messages (29)
msg236158 - (view) Author: John Nagle (nagle) Date: 2015-02-18 01:07
SSL certificate verification fails for "www.verisign.com" when using the cert list from Firefox. Other sites ("google.com", "python.org") verify fine. 

This may be related to a known, and fixed, OpenSSL bug. See:

http://rt.openssl.org/Ticket/Display.html?id=2732&user=guest&pass=guest
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640 

Some versions of OpenSSL are known to be broken for cases where there multiple valid certificate trees.  This happens when one root cert is being phased out in favor of another, and cross-signing is involved.

Python ships with its own copy of OpenSSL on Windows.  Tests
for "www.verisign.com"

Win7, x64:

   Python 2.7.9 with OpenSSL 1.0.1j 15 Oct 2014. FAIL
   Python 3.4.2 with OpenSSL 1.0.1i 6 Aug 2014.  FAIL
   openssl s_client -OpenSSL 1.0.1h 5 Jun 2014   FAIL

Ubuntu 14.04 LTS, x64, using distro's versions of Python:

   Python 2.7.6 - test won't run, needs create_default_context
   Python 3.4.0 with OpenSSL 1.0.1f 6 Jan 2014.  FAIL
   openssl s_client  OpenSSL 1.0.1f 6 Jan 2014   PASS

That's with the same cert file in all cases. The OpenSSL version for Python programs comes from ssl.OPENSSL_VERSION. 

The Linux situation has me puzzled.  On Linux, Python is supposedly using the system version of OpenSSL. The versions match.  Why do Python and the OpenSSL command line client disagree?  Different options passed to OpenSSL by Python?

A simple test program and cert file are attached.  Please try this in your environment.
msg236159 - (view) Author: John Nagle (nagle) Date: 2015-02-18 01:08
Add cert file for testing.  Source of this file is

http://curl.haxx.se/ca/cacert.pem
msg236160 - (view) Author: John Nagle (nagle) Date: 2015-02-18 01:15
To try this with the OpenSSL command line client, use this shell command:

    openssl s_client -connect www.verisign.com:443 -CAfile cacert.pem

This provides more detailed error messages than Python provides.

"verify error:num=20:unable to get local issuer certificate" is the OpenSSL error for "www.verisign.com".  The corresponding Python error is "[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)."
msg236168 - (view) Author: Laura Creighton (lac) Date: 2015-02-18 10:57
I have this problem too.

Debian jessie/sid
 Python 2.7.8 (default, Nov 18 2014, 14:57:17)
 Python 3.4.2 (default, Nov 13 2014, 07:01:52)
msg236318 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-02-20 18:54
> This may be related to a known, and fixed, OpenSSL bug.

Where do you see that the bug is fixed?
msg236321 - (view) Author: Laura Creighton (lac) Date: 2015-02-20 19:02
In https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1014640
it says :

FIX:
Fixed in Ubuntu 14.04 apparently.
Openssl upstream, see http://rt.openssl.org/Ticket/Display.html?id=2732

But I think the person who wrote that launchpad note was mistaken, as
the rt.openssl.org ticket still is marked open when I looked at it.
msg236327 - (view) Author: John Nagle (nagle) Date: 2015-02-20 20:41
The "fix" in Ubuntu was to the Ubuntu certificate store, which is a directory tree with one cert per file, with lots of symbolic links with names based on hashes to express dependencies. Python's SSL isn't using that.  Python is taking in one big text file of SSL certs, with no link structure, and feeding it to OpenSSL.  

This is an option at

 SSLContext.load_verify_locations(cafile=None, capath=None, cadata=None)

I've been testing with "cafile".  "capath" is a path to a set of preprocessed certs laid out like the Ubuntu certificate store.  It may be that the directory parameter works but the single-file parameter does not.  It's possible to create such a directory from a single .pem file by splitting the big file into smaller files (the suggested tool is an "awk" script) and then running "c_rehash", which comes with OpenSSL.  See "https://www.openssl.org/docs/apps/c_rehash.html"  

So I tried a workaround, using Python 3.4.0 and Ubuntu 14.04 LTS.  I broke up "cacert.pem" into one file per cert with the suggested "awk" script, and used "c_rehash" to build all the links, creating a directory suitable for "capath". It didn't help.  Fails for "verisign.com", works for "python.org" and "google.com", just like the original single-file test. The "capath" version did exactly the same thing as the "cafile" version.

Python is definitely reading the cert file or directories; if I try an empty cert file or dir, everything fails, like it should.

Tried the same thing on Win7 x64. Same result. Tried the command line openssl tool using the cert directory. Same results as with the single file on both platforms.

So that's not it. 

A fix to OpenSSL was proposed in 2012, but no action was taken:

http://rt.openssl.org/Ticket/Display.html?id=2732 at
"Wed Jun 13 17:15:04 2012 Arne Becker - Correspondence added".

Any ideas?
msg236506 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-02-24 15:34
> Python's SSL isn't using that.  Python is taking in one big text file 
> of SSL certs, with no link structure, and feeding it to OpenSSL.  

Python's SSL is not "taking" anything:

>>> r = urlopen('https://www.verisign.com')
>>> r.read(10)
b' <!DOCTYPE'

It's only if you feed it that particular CA file that you get the issue:

>>> cafile = 'cacert.pem'
>>> r = urlopen('https://www.verisign.com', cafile=cafile)
[...]
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:600)>


You can *also* feed it a CA directory by using the "CApath" argument (*not* "CAfile").

Now it remains to be seen why "openssl s_client" works with the file nevertheless.
msg236509 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2015-02-24 16:01
John, neither Python nor OpenSSL are shipped with certificates.

Python uses certificates from operating system. We decided against our own certificate store because we wanted to avoid exactly this kind of trouble. If Python can't verify a certificate then you have to update the certificate storage of your OS.

On Linux and BSD Python, curl, wget and most other system tools use the OS's cert store. On Windows Python uses the same store as the IE, Chrome and other apps. Contrary to IE Python doesn't enforce cert store updates.

You can reproduce the problem with curl, too. The first call uses the OS' store, the second overwrite the default store.

$ curl https://www.verisign.com

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp curl https://www.verisign.com
msg236510 - (view) Author: Ian Cordasco (icordasc) * Date: 2015-02-24 16:15
So requests is running into this issue as well (see: https://github.com/kennethreitz/requests/issues/2455, https://github.com/kennethreitz/requests/issues/2456). With the specific code in Cory Benfield's comment (see: https://github.com/kennethreitz/requests/issues/2455#issuecomment-75773677) and the certificate file that requests 2.5.2 used (see: https://github.com/kennethreitz/requests/blob/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem, raw: https://raw.githubusercontent.com/kennethreitz/requests/d8be2473d1a586a3673d728d49e10fd4286e3b0e/requests/cacert.pem) we can reproduce a similar problem on all versions of Python.

At the moment, we're investigating the possibility that it has to do with cross-signed certificates (see: http://openssl.6102.n7.nabble.com/Problems-with-cross-signed-certificates-and-Authority-Key-Info-td52280.html). We have a number of servers that we can reproduce this against and it is not reproducible using openssl s_client which means it is an issue with how Python has written its openssl compatibility layer.
msg236511 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-02-24 16:20
Ok, this is really a bug in the cert bundle provided by requests and Firefox.

With requests 2.5.1:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> ok

With requests 2.5.2:

$ SSL_CERT_DIR=/tmp SSL_CERT_FILE=/tmp openssl s_client -CAfile requests/cacert.pem -connect verisign.com:443

=> Verify return code: 20 (unable to get local issuer certificate)
msg236512 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-02-24 16:22
> and it is not reproducible using openssl s_client

I have determined that s_client is buggy. It will always load the system certs *if and only if* you also pass it a valid custom CA cert (which is the reverse of what's expected).

This is where it happens (in apps/s_client.c):

    if ((!SSL_CTX_load_verify_locations(ctx, CAfile, CApath)) ||
        (!SSL_CTX_set_default_verify_paths(ctx))) {
        /*
         * BIO_printf(bio_err,"error setting default verify locations\n");
         */
        ERR_print_errors(bio_err);
        /* goto end; */
    }

This is why I forced SSL_CERT_* to empty locations in the examples above, so that only the custom CA bundle is used.
msg236513 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2015-02-24 16:31
It appears it's not actually an issue with the CA Bundle, but I don't think it's actually an issue with Python, though Python might be in the best situation to try and fix it...

Basically, it appears that OpenSSL does not look inside the trust root for any certificate served by the server. In this case the sites have a chain that looks like A -> B -> NEW ROOT being served by the server, and NEW ROOT is also signed by OLD ROOT. If I construct the chain being sent from the server so it doens't have NEW ROOT, then everything works, but if the chain being sent from the server has NEW ROOT, then OpenSSL will only trust it if OLD ROOT is in the trust bundle. In this case Mozilla (and requests) has NEW ROOT in the trust bundle but not OLD ROOT, becuase OLD ROOT is a 1024 bit key.
msg236518 - (view) Author: Laura Creighton (lac) Date: 2015-02-24 17:10
Antione closed this, as a not python error, as
if you do not pass a valid certificate to openssl s_client
it will not read the system certificates, which is clearly
utterly surprising and nuts.

The problem, as I see it, is that fixing this clear
absurdity may not fix a different underlying problem.  So this
one may need reopening  when the real error us revealed.  
See if John Nagel's code works ...
msg236520 - (view) Author: Cory Benfield (Lukasa) * Date: 2015-02-24 17:15
The problem specifically is that OpenSSL only uses a *root* in the trust store as an anchor. That means any certificate that is signed by another certificate will not terminate the chain of trust. Browsers do better here, by trusting the entirety of the trust store, regardless of whether or not it's a root certificate.

Donald is correct: this is not really Python's fault, it's OpenSSL's.
msg236548 - (view) Author: John Nagle (nagle) Date: 2015-02-24 21:32
I've reported this as an update to OpenSSL bug #2634. 
(http://rt.openssl.org/Ticket/Display.html?id=2634)
Now we have to follow up there.

This bug should probably be set to "pending", not "closed".  The problem is upstream, but OpenSSL is the Python libraries' choice, not the users'.
Python for Windows ships with its own copy of OpenSSL, so when (if) OpenSSL is fixed, the Python Windows distros will need an update.
msg236976 - (view) Author: Ian Cordasco (icordasc) * Date: 2015-03-01 20:14
So it seems like https://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=3621 includes a fix that we may be able to update Python to use (safely) by default. If we don't then this will continue to be an issue.

Other references:

- https://bugzilla.redhat.com/show_bug.cgi?id=1166614

For now RedHat is keeping the 1024-bit certificates around for backwards compatibility and only because that option isn't set by default.
msg236977 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2015-03-01 20:18
There actually *is* an API that can be set that will cause OpenSSL to use the shortest trust path it can, however it's only available in OpenSSL 1.0.2+ which means it'll solve it for a handful of people but not the bulk of people.
msg236978 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2015-03-01 20:19
I'm attaching a patch that does what Donald suggests.
msg236982 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2015-03-01 21:41
With the patch the flag is always set. Are there any possible side effects? IMHO it's better to add a store_flags property and make the feature optional.
msg236983 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2015-03-01 21:42
It looks like the existing `verify_flags` param is actually the same thing, so we can just use it. That said, I think this should be on by default, I can't think of a scenario you don't want it.
msg236984 - (view) Author: Cory Benfield (Lukasa) * Date: 2015-03-01 21:44
My reading of the OpenSSL issue is that there are no negative side effects from turning this on.
msg237238 - (view) Author: Roundup Robot (python-dev) Date: 2015-03-05 03:11
New changeset 7f64437a707f by Benjamin Peterson in branch '3.4':
enable X509_V_FLAG_TRUSTED_FIRST when possible (closes #23476)
https://hg.python.org/cpython/rev/7f64437a707f

New changeset 37da00170836 by Benjamin Peterson in branch '2.7':
enable X509_V_FLAG_TRUSTED_FIRST when possible (closes #23476)
https://hg.python.org/cpython/rev/37da00170836

New changeset 442e2c357979 by Benjamin Peterson in branch 'default':
merge 3.4 (#23476)
https://hg.python.org/cpython/rev/442e2c357979
msg237253 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-05 09:30
Benjamin, can you please add at least a comment describing why you added the flag? We have enough obscure-looking code in _ssl.c as it is.
msg237254 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-05 09:35
Uh, the comment is already there. I don't know how I missed that. Sorry.
msg237273 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2015-03-05 15:45
The Windows binaries of Python 2.7.9 are compiled with OpenSSL 1.0.1j. The feature is only available in OpenSSL > 1.0.2. The next version of Python must be compiled with 1.0.2 or better. Otherwise the bug pops up again.
msg237288 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2015-03-05 21:03
Issue23593 opened to request Windows and OS X installer OpenSSL updates to 1.0.2
msg237291 - (view) Author: John Nagle (nagle) Date: 2015-03-05 21:06
Will this be applied to the Python 2.7.9 library as well?
msg237292 - (view) Author: Donald Stufft (dstufft) * (Python committer) Date: 2015-03-05 21:07
It was merged to the 2.7 branch, so it'll be released as part of 2.7.10.
History
Date User Action Args
2015-03-20 12:11:45jceasetnosy: + jcea
2015-03-05 21:07:31dstufftsetmessages: + msg237292
2015-03-05 21:06:47naglesetmessages: + msg237291
2015-03-05 21:03:50ned.deilysetnosy: + ned.deily
messages: + msg237288
2015-03-05 15:45:29christian.heimessetmessages: + msg237273
2015-03-05 09:35:33pitrousetmessages: + msg237254
2015-03-05 09:30:05pitrousetmessages: + msg237253
2015-03-05 03:11:56python-devsetstatus: open -> closed

nosy: + python-dev
messages: + msg237238

resolution: fixed
stage: patch review -> resolved
2015-03-05 02:52:53dstufftsetkeywords: + needs review
status: closed -> open
resolution: not a bug -> (no value)
stage: resolved -> patch review
2015-03-01 21:44:59Lukasasetmessages: + msg236984
2015-03-01 21:42:42alexsetmessages: + msg236983
2015-03-01 21:41:30christian.heimessetmessages: + msg236982
2015-03-01 20:19:45alexsetfiles: + store.diff
keywords: + patch
messages: + msg236978
2015-03-01 20:18:22dstufftsetmessages: + msg236977
2015-03-01 20:14:57icordascsetmessages: + msg236976
2015-02-24 21:32:19naglesetmessages: + msg236548
2015-02-24 17:19:34Aaron.Meurersetnosy: + Aaron.Meurer
2015-02-24 17:15:40Lukasasetmessages: + msg236520
2015-02-24 17:10:35lacsetmessages: + msg236518
2015-02-24 16:31:27dstufftsetmessages: + msg236513
2015-02-24 16:23:47pitrousetstatus: open -> closed
resolution: not a bug
stage: resolved
2015-02-24 16:22:57pitrousetmessages: + msg236512
2015-02-24 16:20:37pitrousetmessages: + msg236511
2015-02-24 16:15:54icordascsetnosy: + icordasc
messages: + msg236510
2015-02-24 16:06:19Lukasasetnosy: + Lukasa
2015-02-24 16:02:00christian.heimessetmessages: + msg236509
2015-02-24 15:39:44pitrousetnosy: + janssen, giampaolo.rodola, christian.heimes, alex, dstufft

type: behavior
versions: + Python 3.5
2015-02-24 15:34:38pitrousetmessages: + msg236506
2015-02-20 20:41:57naglesetmessages: + msg236327
2015-02-20 19:02:45lacsetmessages: + msg236321
2015-02-20 18:54:09pitrousetnosy: + pitrou
messages: + msg236318
2015-02-20 18:19:24demian.brechtsetnosy: + demian.brecht
2015-02-18 10:57:22lacsetnosy: + lac
messages: + msg236168
2015-02-18 01:15:36naglesetmessages: + msg236160
2015-02-18 01:09:05naglesetfiles: + cacert.pem

messages: + msg236159
2015-02-18 01:07:17naglecreate