Apple has deprecated use of openssl in OS X due to its unstable API between versions:

"If your app depends on OpenSSL, you should compile OpenSSL yourself and statically link a known version of OpenSSL into your app"

https://developer.apple.com/library/mac/#documentation/security/Conceptual/cryptoservices/GeneralPurposeCrypto/GeneralPurposeCrypto.html

Currently OS X ships with patched versions of libssl 0.9.7 and 0.9.8.  The 32-bit python.org installer links with and dynamically loads 0.9.7 and the 64-/32-bit installer with 0.9.8.

build-installer.py should be enhanced to build and link with its own universal more up-to-date static libssl, as is done for several other OS X-supplied libraries.  Since apparently the openssl upstream builds do not support OS X universal builds, build-installer.py will need to learn how to build each arch separately and lipo them together.

With the current discussion around security issues, are there features in openssl 1.x.x that warrant making this a release blocker for 2.7.4 and 3.2.4?  I should be able to implement and test this over the next few days if so.

Setting to release blocker for release managers' decision.

As you are the MacOSX expert, I'm going to defer your judgement (and/or Ronald's). I don't think the release will be for several days at least, so you should have time to test.

I'm not sure if it is worthwhile to switch right now.  Apple does deprecate the use of OpenSSL, but there version does offer a feature that's not in the default tree: it verifies SSL certificates against the CA list in the system keychain.   

This means that users that verify certificates (cert_reqs=CERT_REQUIRED in the ssl module) could see a regression when they don't specificy a custom CA list. Not having to maintain such a list manually is very convenient.

In the longer run I'd like to try if it is possible to implement the SSL module (and other extensions linking with openssl) using Apple's crypto APIs.

(Note that a clear disadvantage of the latter is that those APIs are "above" the unix layer and likely cause problems when you use fork(2) without exec(2)).

Yes, as we've discussed, using the Apple Crypto APIs would be nice longer-term assuming the compatibility issues can be managed: the set of available APIs appear to have been evolving over the past several OS X releases.  But moving away from openssl seems out of scope for maintenance releases.  The reason why I brought this up now is to try to determine if there are any newer ssl features that may be need to be batteries-included to deal with possible changes to Distutils and/or its users (pip, et al).   

Managing the CA certificates certainly is another issue.  We should investigate what Apple and third-party distributors of openssl on OS X do.  It would be best to be able to use the system-supplied ones since they are actively managed by Apple and can be by the user.

Also, it would be good to know how the Python Windows distribution handles openssl and certificates.  Martin?  Brian?

Using the CA bundle from the OS sounds great, not only for Macs :)

Somewhat coincidentally, Issue17129 addresses the topic of certificate management across multiple platforms.

Replacing openssl by the supported crypto api's is something for 3.4 or even 3.5.

There is a way to keep the current functionality while still shipping a build of openssl: apply the patch that implements the feature to the upstream version when building it (the patch is available on opensource.apple.com, that's how I know that they do this in the first place).

Something that should be tested before this gets merged: what happens when a user installs pyOpenSSL with python 2.7.3 install (linked to system openssl) and then upgrades to 2.7.4 linked to a custom build of openssl without changing pyOpenSSL.  

I wouldn't expect problems when looking at the documentation (there doesn't seem to be a way to transfer SSL state at the C level), and something similar can already happen: python is linked with a fairly old version of OpenSSL, and you get a later version when linking on a newer OSX release (hence a lot of users that download the binary installer and then install pyOpenSSL already have a version mismatch between the two extensions using openssl).

On Windows urllib.request.urlopen("http://www.google.com", cadefault=True) fails with "certificate verify failed". (tested with Python 3.3 64bit)

It doesn't raise an exception on OSX (close to the tip of the default branch), both for http://www.google.com/ and https://www.google.com/

After spending some time on this, I'm downgrading this from release blocker status.  First, no one has yet identified any immediate need for openssl 1.0.x features to support possible PyPI enhancements, which was my original concern.  Second, since the openssl build system does not support OS X universal builds or SDKs and is not autoconf-based, it does not fit well into the current OS X installer build process.  I have a working first cut of building the libs but there is more to do.  Third, there is the open issue of how to make root certs available.  Ronald, I'm probably missing something obvious here but I don't see which Apple patch you are referring to.  Can you elaborate?

There is also the issue of government export restrictions that seems to always come up with crypto software.  AFAICT, as of a couple of years ago, there is no longer any restriction on shipping openssl binaries with any encryption algorithm from the US to any other country.  There are still a few well-known patent issue which seem easy to avoid.  But I am not a lawyer.
 
Unless someone objects, I'm going to treat this as a new feature for now and, once ready, we can re-examine backporting.

See also: issue 15740


A version of OpenSSL as included in some versions of OSX can be downloaded from <http://opensource.apple.com/tarballs/OpenSSL098/>, as mentioned in issue 15740 the versions as included in the most recent OS updates doesn't seem to be there.

I've downloaded OpenSSL098-35.1 and that includes files ./src/crypto/x509/x509_vfy_apple.h and ./src/crypto/x509/x509_vfy_apple.c which implement the behavior I mentioned earlier: first try to verify using the default OpenSSL mechanism, then verify using the TrustEvaluationAgent. 

Now that I look at that code again: we can't extract that code and use it to patch upstream OpenSSL, the TrustEvaluationAgent framework is a private framework and hence off limits.

It is probably possible to reimplement the same feature using public APIs, but that's new development and should be off-limits for a bugfix release (and isn't something that can be done very soon without risking to introduce new bugs in security-related code).

Direct link to the source code I mentioned: http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.c, http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.h


A blog about this feature by the one of the curl developers: http://daniel.haxx.se/blog/2011/11/05/apples-modified-ca-cert-handling-and-curl/


P.S. Apple doesn't exactly make it easy to find this information.

It should be noted that latest OSX Mountain Lion has caused problems for other language libraries too (specifically ruby, which I use at work). Ease the support of correct openssl in OSX may help a long way in all versions of python.

What other problems? Do you have more information on that?

Ronald:

Sorry for the delay in response. I was referring to this:

https://github.com/capistrano/capistrano/issues/250#issuecomment-8055991

" Mountain Lion no longer includes OpenSSL as it uses a different SSL
implementation. So, you need to have RVM install OpenSSL, then
reinstall rubies. This also fixes the SSL error with bundler updating
gems."

The RVM issue is wrong, ML still includes OpenSSL. Apple has deprecated the use of the system install of OpenSSL, but the library and include files are still there.

There are two paths for avoiding the deprecated library: either ship your own build of OpenSSL, or (and that's probably what Apple would prefer) use the Apple specific frameworks (common crypto and/or security transforms). 

The latter has the advantage of using the security infrastructure, such as the CA chain, as provided by Apple, but would require significant code changes in Python, might take even more work to get to work properly on OSX 10.6 or earlier (and cannot work on 10.4), and might cause problems with scripts that use os.fork() because a number of core Apple frameworks won't work properly in the child process and cause a hard crash when they were initialized in the parent and then used in a child.

All in all it would be better to ship a recent version of OpenSSL with the binary installers on OSX.

The one difference between the system openssl and a separately compiled one is that the former can use the CA root from the KeyChain (and uses a private API to do that, as noted earlier).

I just stumbled across a utility that can sync the KeyChain to an OpenSSL CA file: <https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m>, and a blog message at <http://landonf.bikemonkey.org/code/macosx/certsync.20130514.html>

I managed to build Python with OpenSSL 1.0.1e on Mac OS 10.8.4 using build-installer.py script during Europython sprint. I'll attach patches in few days, as I'll try to clean the code a bit. Currently the build works only for intel.

Here is an example output:
test-osx ~ ➤ uname -a
Darwin test-osx.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May  1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
test-osx ~ ➤ arch -i386 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:34)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize
2147483647
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
>>> ^D
test-osx ~ ➤ arch -x86_64 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:35)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize
9223372036854775807
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
>>> ^D
test-osx ~ ➤

Can you please run the unit test to see if it actually works?

 arch -i386 python -m test -unetwork test_ssl

That commands runs the unit test suite and tries to connect to a couple of HTTPS servers, too.

By the way do you know how well supported and up to date /System/Library/OpenSSL/certs/cert.pem is? The file suppose to contain all CA certs.

Chrys, /System/Library/OpenSSL/certs/cert.pem is empty on current OS X systems. As mentioned above, cert management is handled through Apple APIs.  See http://developer.apple.com/library/ios/#DOCUMENTATION/Security/Conceptual/CertKeyTrustProgGuide/01introduction/introduction.html

The file not even empty, it doesn't even exist in default installations.

As Ned mentioned the CA roots on OSX are stored in a system database (the keychain). The situation is more or less the same as on Windows: their either needs to be code that queries the system keychain to get the CA roots, or we need to use some other CA list (like the mozilla one). 

I'd prefer the former because that's easier for the end user (add the company-wide CA cert to the system database to configure it system wide instead of hunting down where every app stores it CA list).

One glitch: a lot of Apple API's above the Unix layer are unsafe when used in a process started with os.fork (without calling execv), I don't know yet if the keychain APIs are also affected by this.

How about we include certsync [1] as an external program then? It's not as elegant as an internal API but it's not going to cause trouble with forking servers.

The first time a root CA cert is require, some new code in Python's SSL module runs certsync and grabs the list of PEM encoded certs from its stdout. The output is cached in a module variable and loaded with the new features from #16487 and #18138.

How does that sound to you?

[1] https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m

The other way to solve problem with CA certs may be this script[1] that can be used to generate pem files from mozilla CA cert bundle[2] during compilation or installation.

[1]: http://curl.haxx.se/docs/caextract.html
[2]: http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt

Mateusz,
we would like to avoid that path and use the operating system's root CAs list when the OS provides one. It gives better user experience and the OS' root CA list gets updates through the OS.

I finally managed to find some time to clean up the script I used to compile multiarch openssl.

The diff is available at https://bitbucket.org/_mlen/cpython/commits/319f10362eb4b947b12750a6eb66fb622bbb0079

For 10.8 I had to comment out Tk related part and use the command below to build:
./build-installer.py --sdk-path=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/ --third-party=/Users/m/Documents/Code/sources --universal-archs=intel --dep-target=10.8

The test results are below:
test-osx ➤ arch -i386 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.
test-osx ➤ arch -x86_64 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.

Update: the MacPorts certsync daemon has matured and is now included as
an optional MacPorts port.

It's not a perfect solution as noted in the macports-devel thread:

>>The only catch is that custom added certificates or trust anchors need
>>to be in the system keychain to be picked up by certsync by default.
>Yeah, this was an unfortunate trade-off; since certsync is a system-wide 
>daemon, and the resulting CA certs file is also system-wide, it seemed to
>be the most appropriate course of action. Most of the alternatives involve
>patching OpenSSL and some of the software that depends on it, which is a
>road I'm personally wary of committing to.

http://comments.gmane.org/gmane.os.apple.macports.devel/22653

It works by registering a launchd agent that is run whenever any of the
system keychain files or trust setting files are modified.  That raises the
issues of when and how a Python install should register the agent (will
likely need admin/root privileges to do that) and how to delete the agent
(we currently don't have a formal uninstall procedure on OS X).  It would
be easier to manage these things with the binary installer-based Pythons,
as provided by python.org, in which case Pythons built by users from source
would still use the deprecated Apple-supplied libssl and libcrypto.  But
I'd like to separate out the building of third-party libraries, like
openssl, from the installer build process so that user source builds can
take advantage of features like this.

Thanks to Hynek we were able to dig deeper into Apple's modifications. OpenSSL on OSX uses TEA (TrustEvaluationAgent) to verify cert chains. TEA is pretty much undocumented on the internet but perhaps we can use it to verify certs with OpenSSL 1.x, too?

http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.c

Crys, as Ronald noted above: "Now that I look at that code again: we can't extract that code and use it to patch upstream OpenSSL, the TrustEvaluationAgent framework is a private framework and hence off limits."  It doesn't seem like a good idea to be trying to base security on a private, undocumented framework and one that can change from OS X release to OS X release: our binary installers for OS X are designed to support multiple OS X versions.  I think the certsync approach is safer and more robust.  The other approach would be to directly use Apple's crypto APIs rather than OpenSSL but that would be a lot of work and a lot of testing and would also be more coupled to specific OS X releases.

AFAIK OpenSSL has hooks that can be called when a certificate needs to be validated. If I my memory is correct this could be used to validate certificates using a public API (basically doing the same as Apple's patch, but using public APIs for the system and OpenSSL).

This has one significant risk though: as we've found at a couple of times (such as with the _scproxy extension) Apple's API don't necessary play along nicely when you use execv without fork or fork without execv :-(. I have no idea if Apple's preferred crypto APIs suffer from this problem.

The link below contains a script for building fat binaries for openssl. There's nothing surprising in the script, just building multiple times and then merging the result using lipo.

https://gist.github.com/foozmeat/5154962

BTW. I'm not proposing to use that script, at most it could be used for inspiration to enhance the build-installer script.

New changeset 31f506f4e2d2 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/31f506f4e2d2

New changeset 2b5fa8651bd0 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/2b5fa8651bd0

New changeset 288b03b5c60d by Ned Deily in branch '3.4':
Issue #17128: Use private version of OpenSSL for 3.x OS X 10.5+ installers.
https://hg.python.org/cpython/rev/288b03b5c60d

New changeset 1c249d0cab5d by Ned Deily in branch 'default':
Issue #17128: Merge / update from 3.4
https://hg.python.org/cpython/rev/1c249d0cab5d

New changeset 961f988eaa6a by Ned Deily in branch '2.7':
Use rtf format files for legacy OS X installer builds.
https://hg.python.org/cpython/rev/961f988eaa6a

The OpenSSL included with OS X, still at 0.9.8, has become very dated and a growing number of servers (including openssl.org) now fail the handshake because they no longer support any of the protocols and ciphers included with that build. It is reaching the point where for some projects the system OpenSSL on OS X is no longer viable to use.

Here's a ticket for a Python project showing an example of this problem:

https://github.com/kivy/kivy-ios/issues/198

I have run into this issue myself on this and a couple other projects I am working on.

Would it be possible to have the fix for 10.5 builds get applied to all the Python OS X installers, perhaps for 3.6, and ideally also for 2.7 if there are plans to make a new release for that?

New changeset bfd0a73cf907 by Ned Deily in branch 'default':
Issue #17128: Build OS X installer for 3.6 with private copy of OpenSSL.
https://hg.python.org/cpython/rev/bfd0a73cf907

Hello,

I see that the official Python.org OSX 10.6+ installers are still linking with OSX outdated OpenSSL (0.9.8zh 14 Jan 2016; I'm using macOS High Sierra 10.13.2).

In the installer's README, they motivates this because:

> Apple's 0.9.8 version includes an important additional feature: if a certificate cannot be verified using the manually administered certificates in /System/Library/OpenSSL, the certificates managed by the system security framework in the user and system keychains are also consulted (using Apple private APIs)

However, because of this outdated OpenSSL version, I cannot use pip to install from the TestPyPI server https://test.pypi.org.

When I try (even with --trusted-host) I get this error:

```
$ pip install --trusted-host --index-url https://test.pypi.org/simple afdko
Collecting https://test.pypi.org/simple
Exception:
Traceback (most recent call last):
...
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:661)
```

A similar issue on pypa/pip repository was closed as

> There's no actionable item here. People with old versions of openssl that don't support sha1 SSL certificates need to upgrade or else they are insecure. If they wish to be insecure they can continue using pip 1.2

https://github.com/pypa/pip/issues/829#issuecomment-20931050

~~~

Well, I find it particularly odd that the official binary distribution for the latest Python 2.7.14 has a broken (or insecure) _ssl module, even when running the latest macOS version.

Of course, using pyenv or homebrew fixes the problem (as they require and link with the latest openssl 1.0.2), but I would like to recommend installing python from the official binaries to my less technically-skilled colleagues of mine.

Please consider embedding the latest openssl on the 10.6+ installers like you already do on 10.5 32bit ones and the Windows ones, thank you.

Cosimo Lupo

Ned, is this issue still relevant? It targets 3.5 and hasn't seen much activity since 2016.

New changeset 93c91ac8f67ef0819ddacdcca12ef0ae88e5802e by Ned Deily in branch '2.7':
[2.7] Fix errant NEWS item: bpo-19019 -> bpo-17128 (GH-6470)
https://github.com/python/cpython/commit/93c91ac8f67ef0819ddacdcca12ef0ae88e5802e

As of 2.7.15 (finally!) all current python.org macOS installers now provide their own copies of OpenSSL: 1.0.2x for 3.6.x and 2.7.15. 1.1.0x for the upcoming 3.7.0 release.  Until a better solution for root certificates is implemented, all of the installers also now provide a "Install Certificates.command', a script which will download and install the latest third-party `certifi` bundle of certificates from PyPI and configure the ssl module to use them.  Declaring this issue resolved.  Thanks everyone!