classification
Title: Python SSL stack doesn't have a default CA Store
Type: security Stage:
Components: Library (Lib) Versions: Python 3.4
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, Dima.Tisnek, barry, benjamin.peterson, christian.heimes, dstufft, eric.araujo, fweimer, icordasc, jcea, koobs, lnussel, loewis, naif, pitrou
Priority: normal Keywords:

Created on 2011-12-23 10:18 by naif, last changed 2015-01-02 05:13 by benjamin.peterson. This issue is now closed.

Messages (13)
msg150142 - (view) Author: naif (naif) Date: 2011-12-23 10:18
For the certificate store:

Can we eventually agree to bind a default CA-store to a Mozilla verified one?
Mozilla in handling Firefox does a great job in keeping CA-store up-to-date.

Integrating default mozilla CA-store with Python builds could be a nice way, it's just a matter of integrating into the build-system the download/fetching of default Mozilla store.

At least the language base it's default on a trusted entity to manage, cross-platform, the CA-store for TLS/SSL.

The mainteinance of the CA-store would be delegated to Mozilla that has been demonstrated to be independent and very security conscious, removing dirty CA-store (like Diginotar after Iranian compromise).

That way 90% of case of of SSL/TLS certificate validation will be managed and by default it would be possible to enable secure SSL/TLS client checking like described in http://bugs.python.org/issue13647 .
msg150147 - (view) Author: naif (naif) Date: 2011-12-23 10:39
Mozilla CA are available on:

https://www.mozilla.org/projects/security/certs/

The warranty and security process of Mozilla handling of SSL CA root certs is described on:

https://wiki.mozilla.org/CA

I think that Python language could reasonably base it's default root CA on the Mozilla ones that are the most recognized for security and transparency in the world.
msg150187 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2011-12-23 17:29
I'm not sure Python should be in the business of distributing CA certificates. I think it's better left to the application or Linux distribution.
msg181414 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-05 03:50
I propose to change the scope of this request to: ssl module should provide a way to access the OS CA bundle.
msg181460 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-05 16:44
Copy of a message by Christian Heimes on a duplicate report:
  
For effective SSL server cert validation a bundle of trustworthy CA certs is required. Most system ship such a bundle but it's not always possible to access the bundle from Python / OpenSSL. Windows and Mac OS X come into my mind. wget and curl ship a copy of Mozilla's CA cert bundle.

The site http://curl.haxx.se/docs/caextract.html explains how to extract the CA certs in PEM format. I suggest that we ship the CA bundle with Python and use a lookup chain:

- user defined path to a cacert directory or cacert.pem file

- cacert directory or PEM file in the user's home directory: 
  cacertdir = os.path.join(site.USER_SITE, os.pardir, "cacert")
  cacertfile = os.path.join(site.USER_SITE, os.pardir, "cacert.pem")

- system's ca cert directory (/etc/ssl/certs on Linux)

- CA cert bundle shipped with the Python installation.
msg181489 - (view) Author: Ian Cordasco (icordasc) * Date: 2013-02-05 22:13
Éric's suggestion is also implemented in python-requests if I remember correctly. It allows for user-specified PEM files and tries to find the operating system bundle. This would be a wonderful inclusion in the standard library.
msg181589 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-07 10:00
> Éric's suggestion is also implemented in python-requests if I remember 
> correctly. It allows for user-specified PEM files and tries to find the 
> operating system bundle. This would be a wonderful inclusion in the
> standard library.

Aren't load_verify_locations() and set_default_verify_paths() sufficient?

http://docs.python.org/dev/library/ssl.html#ssl.SSLContext.load_verify_locations
msg192601 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-07 23:35
I think we can improve the situation with shipping our own CA certs. Almost every operating system or distribution comes with a set of CA certs.

I lots of Linux distributions and most BSD systems. All except FreeBSD install CA certs by default. A fresh FreeBSD systems doesn't have certs but ``pkg_add -r ca-root-nss`` fixes that. At least some versions of SuSE don't have a cafile but rather a capath directory. On Windows #17134 and #16487 are going to allow us to use Windows' cert store through crypt32.dll.

Here is a full list:

cert_paths = [
    # Debian, Ubuntu, Arch, SuSE
    # NetBSD (security/mozilla-rootcerts)
    "/etc/ssl/certs/",
    # Debian, Ubuntu, Arch: maintained by update-ca-certificates
    "/etc/ssl/certs/ca-certificates.crt",
    # Red Hat 5+, Fedora, Centos
    "/etc/pki/tls/certs/ca-bundle.crt",
    # Red Hat 4
    "/usr/share/ssl/certs/ca-bundle.crt",
    # FreeBSD (security/ca-root-nss package)
    "/usr/local/share/certs/ca-root-nss.crt",
    # FreeBSD (deprecated security/ca-root package, removed 2008)
    "/usr/local/share/certs/ca-root.crt",
    # FreeBSD (optional symlink)
    # OpenBSD
    "/etc/ssl/cert.pem",
    # Mac OS X
    "/System/Library/OpenSSL/certs/cert.pem",
    ]

I'd like to add the list to our ssl.py and add an API to check and load certs from that files, directories and other places (Windows).
msg192639 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-07-08 11:56
> I think we can improve the situation with shipping our own CA certs.
> Almost every operating system or distribution comes with a set of CA
> certs.

Why would we ship our own CA certs if every OS comes with CA certs?

> I lots of Linux distributions and most BSD systems. All except
> FreeBSD install CA certs by default. A fresh FreeBSD systems doesn't
> have certs but ``pkg_add -r ca-root-nss`` fixes that.

Kudos to FreeBSD.
Anyway, isn't SSLContext.set_default_verify_paths() enough already?

> Here is a full list: [snip full list]

I don't think it's a good idea to maintain a list of hard-coded
paths in Python: it's not manageable, and it will always become
outdated. If there was a widely-respected standard (e.g. in FHS or
LSB), things would be a lot better.
msg192652 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2013-07-08 13:50
On Jul 08, 2013, at 11:56 AM, Antoine Pitrou wrote:

>I don't think it's a good idea to maintain a list of hard-coded
>paths in Python: it's not manageable, and it will always become
>outdated. If there was a widely-respected standard (e.g. in FHS or
>LSB), things would be a lot better.

I agree.  I don't think we should be shipping certs, but if we do, then it
must be possible and easy for e.g. Linux distros to override.  Linux distros
are already managing certs through their normal and security updates, so it's
a burden to also have to do so for Python.  I think this analogous to shipping
other types of external databases, e.g. timezones, etc.
msg204649 - (view) Author: Dima Tisnek (Dima.Tisnek) Date: 2013-11-28 13:08
re: cert_paths = [...]

This approach is rather problematic, there's no guarantee that a path trusted on one system is trusted on another.

I saw this in setuptools branch, where it does:

for path in cert_path:
    if os.path.exists(path)
        return path

Let's say you're user1 on osx and your native true path is "/System/Library/OpenSSL/certs/cert.pem", can you guarantee that someone else, user2, cannot sneak their hacked files into "/etc/pki/" (presumably missing altogether) or "/usr/local/share/"?

Because if user2 can do that, suddenly user1 verifies all traffic against hacked ca list.
msg204650 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-11-28 13:18
All these paths are on directories that are supposed to be read-only for untrusted users. You can't protect yourself against a malicious admin anyway. For Python 3.4 the ssl module uses the cert path that are configured with OpenSSL. The paths and configuration are outside our control.
msg233307 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2015-01-02 05:13
I don't think we're planning to distribute our own store of certs.
History
Date User Action Args
2015-01-02 05:13:44benjamin.petersonsetstatus: open -> closed
resolution: works for me
messages: + msg233307
2015-01-02 05:12:12koobssetnosy: + koobs
2013-11-28 13:18:46christian.heimessetmessages: + msg204650
2013-11-28 13:08:11Dima.Tisneksetnosy: + Dima.Tisnek
messages: + msg204649
2013-08-26 14:31:13lnusselsetnosy: + lnussel
2013-08-24 22:42:36dstufftsetnosy: + dstufft
2013-07-08 13:50:57barrysetmessages: + msg192652
2013-07-08 11:56:42pitrousetmessages: + msg192639
2013-07-07 23:35:55christian.heimessetnosy: + christian.heimes
messages: + msg192601
2013-06-03 18:16:23barrysetnosy: + barry
2013-03-18 23:41:32Arfreversetnosy: + Arfrever
2013-03-08 08:48:59fweimersetnosy: + fweimer
2013-02-07 10:00:09pitrousetmessages: + msg181589
2013-02-05 22:13:54icordascsetnosy: + icordasc
messages: + msg181489
2013-02-05 16:44:06eric.araujosetmessages: + msg181460
2013-02-05 16:39:55eric.araujolinkissue17129 superseder
2013-02-05 03:50:23eric.araujosetmessages: + msg181414
versions: + Python 3.4, - Python 3.3
2011-12-30 21:27:46eric.araujosetnosy: + pitrou
2011-12-30 21:27:34eric.araujosetnosy: + loewis, eric.araujo

versions: - Python 2.6, Python 3.1, Python 2.7, Python 3.2, Python 3.4
2011-12-23 17:29:46benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg150187
2011-12-23 16:54:33jceasetnosy: + jcea
2011-12-23 10:39:58naifsetmessages: + msg150147
2011-12-23 10:20:30naifsettype: security
2011-12-23 10:18:53naifcreate