classification
Title: OS X system openssl deprecated - installer should build local libssl
Type: Stage: needs patch
Components: Build, macOS, SSL Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: ned.deily Nosy List: Kevin Ollivier, benjamin.peterson, christian.heimes, dilettant, dstufft, eric.araujo, esc24, georg.brandl, larry, loewis, mlen, ned.deily, orsenthil, piotr.dobrogost, pitrou, python-dev, ronaldoussoren
Priority: high Keywords:

Created on 2013-02-04 18:31 by ned.deily, last changed 2016-09-15 07:48 by christian.heimes.

Repositories containing patches
https://bitbucket.org/_mlen/cpython
Messages (34)
msg181366 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-04 18:31
Apple has deprecated use of openssl in OS X due to its unstable API between versions:

"If your app depends on OpenSSL, you should compile OpenSSL yourself and statically link a known version of OpenSSL into your app"

https://developer.apple.com/library/mac/#documentation/security/Conceptual/cryptoservices/GeneralPurposeCrypto/GeneralPurposeCrypto.html

Currently OS X ships with patched versions of libssl 0.9.7 and 0.9.8.  The 32-bit python.org installer links with and dynamically loads 0.9.7 and the 64-/32-bit installer with 0.9.8.

build-installer.py should be enhanced to build and link with its own universal more up-to-date static libssl, as is done for several other OS X-supplied libraries.  Since apparently the openssl upstream builds do not support OS X universal builds, build-installer.py will need to learn how to build each arch separately and lipo them together.

With the current discussion around security issues, are there features in openssl 1.x.x that warrant making this a release blocker for 2.7.4 and 3.2.4?  I should be able to implement and test this over the next few days if so.

Setting to release blocker for release managers' decision.
msg181389 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-02-04 21:27
As you are the MacOSX expert, I'm going to defer your judgement (and/or Ronald's). I don't think the release will be for several days at least, so you should have time to test.
msg181392 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-04 21:46
I'm not sure if it is worthwhile to switch right now.  Apple does deprecate the use of OpenSSL, but there version does offer a feature that's not in the default tree: it verifies SSL certificates against the CA list in the system keychain.   

This means that users that verify certificates (cert_reqs=CERT_REQUIRED in the ssl module) could see a regression when they don't specificy a custom CA list. Not having to maintain such a list manually is very convenient.

In the longer run I'd like to try if it is possible to implement the SSL module (and other extensions linking with openssl) using Apple's crypto APIs.

(Note that a clear disadvantage of the latter is that those APIs are "above" the unix layer and likely cause problems when you use fork(2) without exec(2)).
msg181397 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-04 22:48
Yes, as we've discussed, using the Apple Crypto APIs would be nice longer-term assuming the compatibility issues can be managed: the set of available APIs appear to have been evolving over the past several OS X releases.  But moving away from openssl seems out of scope for maintenance releases.  The reason why I brought this up now is to try to determine if there are any newer ssl features that may be need to be batteries-included to deal with possible changes to Distutils and/or its users (pip, et al).   

Managing the CA certificates certainly is another issue.  We should investigate what Apple and third-party distributors of openssl on OS X do.  It would be best to be able to use the system-supplied ones since they are actively managed by Apple and can be by the user.

Also, it would be good to know how the Python Windows distribution handles openssl and certificates.  Martin?  Brian?
msg181408 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-02-05 03:08
Using the CA bundle from the OS sounds great, not only for Macs :)
msg181410 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-05 03:18
Somewhat coincidentally, Issue17129 addresses the topic of certificate management across multiple platforms.
msg181420 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-05 06:53
Replacing openssl by the supported crypto api's is something for 3.4 or even 3.5.

There is a way to keep the current functionality while still shipping a build of openssl: apply the patch that implements the feature to the upstream version when building it (the patch is available on opensource.apple.com, that's how I know that they do this in the first place).

Something that should be tested before this gets merged: what happens when a user installs pyOpenSSL with python 2.7.3 install (linked to system openssl) and then upgrades to 2.7.4 linked to a custom build of openssl without changing pyOpenSSL.  

I wouldn't expect problems when looking at the documentation (there doesn't seem to be a way to transfer SSL state at the C level), and something similar can already happen: python is linked with a fairly old version of OpenSSL, and you get a later version when linking on a newer OSX release (hence a lot of users that download the binary installer and then install pyOpenSSL already have a version mismatch between the two extensions using openssl).
msg181433 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-02-05 13:14
On Windows urllib.request.urlopen("http://www.google.com", cadefault=True) fails with "certificate verify failed". (tested with Python 3.3 64bit)
msg181435 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-05 13:19
It doesn't raise an exception on OSX (close to the tip of the default branch), both for http://www.google.com/ and https://www.google.com/
msg181663 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-02-08 08:18
After spending some time on this, I'm downgrading this from release blocker status.  First, no one has yet identified any immediate need for openssl 1.0.x features to support possible PyPI enhancements, which was my original concern.  Second, since the openssl build system does not support OS X universal builds or SDKs and is not autoconf-based, it does not fit well into the current OS X installer build process.  I have a working first cut of building the libs but there is more to do.  Third, there is the open issue of how to make root certs available.  Ronald, I'm probably missing something obvious here but I don't see which Apple patch you are referring to.  Can you elaborate?

There is also the issue of government export restrictions that seems to always come up with crypto software.  AFAICT, as of a couple of years ago, there is no longer any restriction on shipping openssl binaries with any encryption algorithm from the US to any other country.  There are still a few well-known patent issue which seem easy to avoid.  But I am not a lawyer.
 
Unless someone objects, I'm going to treat this as a new feature for now and, once ready, we can re-examine backporting.
msg181664 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-08 09:34
See also: issue 15740


A version of OpenSSL as included in some versions of OSX can be downloaded from <http://opensource.apple.com/tarballs/OpenSSL098/>, as mentioned in issue 15740 the versions as included in the most recent OS updates doesn't seem to be there.

I've downloaded OpenSSL098-35.1 and that includes files ./src/crypto/x509/x509_vfy_apple.h and ./src/crypto/x509/x509_vfy_apple.c which implement the behavior I mentioned earlier: first try to verify using the default OpenSSL mechanism, then verify using the TrustEvaluationAgent. 

Now that I look at that code again: we can't extract that code and use it to patch upstream OpenSSL, the TrustEvaluationAgent framework is a private framework and hence off limits.

It is probably possible to reimplement the same feature using public APIs, but that's new development and should be off-limits for a bugfix release (and isn't something that can be done very soon without risking to introduce new bugs in security-related code).

Direct link to the source code I mentioned: http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.c, http://opensource.apple.com/source/OpenSSL098/OpenSSL098-32/src/crypto/x509/x509_vfy_apple.h


A blog about this feature by the one of the curl developers: http://daniel.haxx.se/blog/2011/11/05/apples-modified-ca-cert-handling-and-curl/


P.S. Apple doesn't exactly make it easy to find this information.
msg181850 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2013-02-10 20:26
It should be noted that latest OSX Mountain Lion has caused problems for other language libraries too (specifically ruby, which I use at work). Ease the support of correct openssl in OSX may help a long way in all versions of python.
msg181880 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-02-11 07:59
What other problems? Do you have more information on that?
msg183512 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2013-03-05 07:12
Ronald:

Sorry for the delay in response. I was referring to this:

https://github.com/capistrano/capistrano/issues/250#issuecomment-8055991

" Mountain Lion no longer includes OpenSSL as it uses a different SSL
implementation. So, you need to have RVM install OpenSSL, then
reinstall rubies. This also fixes the SSL error with bundler updating
gems."
msg189204 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-05-14 09:16
The RVM issue is wrong, ML still includes OpenSSL. Apple has deprecated the use of the system install of OpenSSL, but the library and include files are still there.

There are two paths for avoiding the deprecated library: either ship your own build of OpenSSL, or (and that's probably what Apple would prefer) use the Apple specific frameworks (common crypto and/or security transforms). 

The latter has the advantage of using the security infrastructure, such as the CA chain, as provided by Apple, but would require significant code changes in Python, might take even more work to get to work properly on OSX 10.6 or earlier (and cannot work on 10.4), and might cause problems with scripts that use os.fork() because a number of core Apple frameworks won't work properly in the child process and cause a hard crash when they were initialized in the parent and then used in a child.

All in all it would be better to ship a recent version of OpenSSL with the binary installers on OSX.
msg189244 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-05-14 18:41
The one difference between the system openssl and a separately compiled one is that the former can use the CA root from the KeyChain (and uses a private API to do that, as noted earlier).

I just stumbled across a utility that can sync the KeyChain to an OpenSSL CA file: <https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m>, and a blog message at <http://landonf.bikemonkey.org/code/macosx/certsync.20130514.html>
msg192600 - (view) Author: Mateusz Lenik (mlen) * Date: 2013-07-07 22:43
I managed to build Python with OpenSSL 1.0.1e on Mac OS 10.8.4 using build-installer.py script during Europython sprint. I'll attach patches in few days, as I'll try to clean the code a bit. Currently the build works only for intel.

Here is an example output:
test-osx ~ ➤ uname -a
Darwin test-osx.local 12.4.0 Darwin Kernel Version 12.4.0: Wed May  1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
test-osx ~ ➤ arch -i386 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:34)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize
2147483647
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
>>> ^D
test-osx ~ ➤ arch -x86_64 python3
Python 3.4.0a0 (default, Jul  7 2013, 17:12:35)
[GCC 4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.28)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.maxsize
9223372036854775807
>>> import ssl
>>> ssl.OPENSSL_VERSION
'OpenSSL 1.0.1e 11 Feb 2013'
>>> ^D
test-osx ~ ➤
msg192602 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-07 23:45
Can you please run the unit test to see if it actually works?

 arch -i386 python -m test -unetwork test_ssl

That commands runs the unit test suite and tries to connect to a couple of HTTPS servers, too.

By the way do you know how well supported and up to date /System/Library/OpenSSL/certs/cert.pem is? The file suppose to contain all CA certs.
msg192604 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2013-07-08 00:18
Chrys, /System/Library/OpenSSL/certs/cert.pem is empty on current OS X systems. As mentioned above, cert management is handled through Apple APIs.  See http://developer.apple.com/library/ios/#DOCUMENTATION/Security/Conceptual/CertKeyTrustProgGuide/01introduction/introduction.html
msg192617 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2013-07-08 06:15
The file not even empty, it doesn't even exist in default installations.

As Ned mentioned the CA roots on OSX are stored in a system database (the keychain). The situation is more or less the same as on Windows: their either needs to be code that queries the system keychain to get the CA roots, or we need to use some other CA list (like the mozilla one). 

I'd prefer the former because that's easier for the end user (add the company-wide CA cert to the system database to configure it system wide instead of hunting down where every app stores it CA list).

One glitch: a lot of Apple API's above the Unix layer are unsafe when used in a process started with os.fork (without calling execv), I don't know yet if the keychain APIs are also affected by this.
msg192634 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-08 11:48
How about we include certsync [1] as an external program then? It's not as elegant as an internal API but it's not going to cause trouble with forking servers.

The first time a root CA cert is require, some new code in Python's SSL module runs certsync and grabs the list of PEM encoded certs from its stdout. The output is cached in a module variable and loaded with the new features from #16487 and #18138.

How does that sound to you?

[1] https://svn.macports.org/repository/macports/trunk/dports/security/certsync/files/certsync.m
msg192636 - (view) Author: Mateusz Lenik (mlen) * Date: 2013-07-08 11:54
The other way to solve problem with CA certs may be this script[1] that can be used to generate pem files from mozilla CA cert bundle[2] during compilation or installation.

[1]: http://curl.haxx.se/docs/caextract.html
[2]: http://mxr.mozilla.org/mozilla/source/security/nss/lib/ckfw/builtins/certdata.txt
msg192643 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-08 12:10
Mateusz,
we would like to avoid that path and use the operating system's root CAs list when the OS provides one. It gives better user experience and the OS' root CA list gets updates through the OS.
msg193461 - (view) Author: Mateusz Lenik (mlen) * Date: 2013-07-21 18:30
I finally managed to find some time to clean up the script I used to compile multiarch openssl.

The diff is available at https://bitbucket.org/_mlen/cpython/commits/319f10362eb4b947b12750a6eb66fb622bbb0079

For 10.8 I had to comment out Tk related part and use the command below to build:
./build-installer.py --sdk-path=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk/ --third-party=/Users/m/Documents/Code/sources --universal-archs=intel --dep-target=10.8

The test results are below:
test-osx ➤ arch -i386 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.
test-osx ➤ arch -x86_64 python3 -m test -unetwork test_ssl
[1/1] test_ssl
Resource 'ipv6.google.com' is not available
1 test OK.
msg209814 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-01-31 19:28
Update: the MacPorts certsync daemon has matured and is now included as
an optional MacPorts port.

It's not a perfect solution as noted in the macports-devel thread:

>>The only catch is that custom added certificates or trust anchors need
>>to be in the system keychain to be picked up by certsync by default.
>Yeah, this was an unfortunate trade-off; since certsync is a system-wide 
>daemon, and the resulting CA certs file is also system-wide, it seemed to
>be the most appropriate course of action. Most of the alternatives involve
>patching OpenSSL and some of the software that depends on it, which is a
>road I'm personally wary of committing to.

http://comments.gmane.org/gmane.os.apple.macports.devel/22653

It works by registering a launchd agent that is run whenever any of the
system keychain files or trust setting files are modified.  That raises the
issues of when and how a Python install should register the agent (will
likely need admin/root privileges to do that) and how to delete the agent
(we currently don't have a formal uninstall procedure on OS X).  It would
be easier to manage these things with the binary installer-based Pythons,
as provided by python.org, in which case Pythons built by users from source
would still use the deprecated Apple-supplied libssl and libcrypto.  But
I'd like to separate out the building of third-party libraries, like
openssl, from the installer build process so that user source builds can
take advantage of features like this.
msg212412 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2014-02-28 08:09
Thanks to Hynek we were able to dig deeper into Apple's modifications. OpenSSL on OSX uses TEA (TrustEvaluationAgent) to verify cert chains. TEA is pretty much undocumented on the internet but perhaps we can use it to verify certs with OpenSSL 1.x, too?

http://opensource.apple.com/source/OpenSSL098/OpenSSL098-35.1/src/crypto/x509/x509_vfy_apple.c
msg212414 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-02-28 08:28
Crys, as Ronald noted above: "Now that I look at that code again: we can't extract that code and use it to patch upstream OpenSSL, the TrustEvaluationAgent framework is a private framework and hence off limits."  It doesn't seem like a good idea to be trying to base security on a private, undocumented framework and one that can change from OS X release to OS X release: our binary installers for OS X are designed to support multiple OS X versions.  I think the certsync approach is safer and more robust.  The other approach would be to directly use Apple's crypto APIs rather than OpenSSL but that would be a lot of work and a lot of testing and would also be more coupled to specific OS X releases.
msg212428 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-02-28 13:23
AFAIK OpenSSL has hooks that can be called when a certificate needs to be validated. If I my memory is correct this could be used to validate certificates using a public API (basically doing the same as Apple's patch, but using public APIs for the system and OpenSSL).

This has one significant risk though: as we've found at a couple of times (such as with the _scproxy extension) Apple's API don't necessary play along nicely when you use execv without fork or fork without execv :-(. I have no idea if Apple's preferred crypto APIs suffer from this problem.
msg214333 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2014-03-21 08:26
The link below contains a script for building fat binaries for openssl. There's nothing surprising in the script, just building multiple times and then merging the result using lipo.

https://gist.github.com/foozmeat/5154962

BTW. I'm not proposing to use that script, at most it could be used for inspiration to enhance the build-installer script.
msg232415 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-10 08:11
New changeset 31f506f4e2d2 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/31f506f4e2d2
msg232433 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-10 16:11
New changeset 2b5fa8651bd0 by Ned Deily in branch '2.7':
Issue #17128: Use private version of OpenSSL for 2.7.9 OS X 10.5+ installer.
https://hg.python.org/cpython/rev/2b5fa8651bd0
msg232599 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-13 08:35
New changeset 288b03b5c60d by Ned Deily in branch '3.4':
Issue #17128: Use private version of OpenSSL for 3.x OS X 10.5+ installers.
https://hg.python.org/cpython/rev/288b03b5c60d

New changeset 1c249d0cab5d by Ned Deily in branch 'default':
Issue #17128: Merge / update from 3.4
https://hg.python.org/cpython/rev/1c249d0cab5d

New changeset 961f988eaa6a by Ned Deily in branch '2.7':
Use rtf format files for legacy OS X installer builds.
https://hg.python.org/cpython/rev/961f988eaa6a
msg269701 - (view) Author: Kevin Ollivier (Kevin Ollivier) Date: 2016-07-01 22:09
The OpenSSL included with OS X, still at 0.9.8, has become very dated and a growing number of servers (including openssl.org) now fail the handshake because they no longer support any of the protocols and ciphers included with that build. It is reaching the point where for some projects the system OpenSSL on OS X is no longer viable to use.

Here's a ticket for a Python project showing an example of this problem:

https://github.com/kivy/kivy-ios/issues/198

I have run into this issue myself on this and a couple other projects I am working on.

Would it be possible to have the fix for 10.5 builds get applied to all the Python OS X installers, perhaps for 3.6, and ideally also for 2.7 if there are plans to make a new release for that?
msg276043 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-12 13:39
New changeset bfd0a73cf907 by Ned Deily in branch 'default':
Issue #17128: Build OS X installer for 3.6 with private copy of OpenSSL.
https://hg.python.org/cpython/rev/bfd0a73cf907
History
Date User Action Args
2016-09-15 07:48:48christian.heimessetcomponents: + SSL
2016-09-12 13:39:44python-devsetmessages: + msg276043
2016-07-05 14:01:30brian.curtinsetnosy: - brian.curtin
2016-07-01 22:09:46Kevin Olliviersetnosy: + Kevin Ollivier
messages: + msg269701
2014-12-13 08:35:32python-devsetmessages: + msg232599
2014-12-10 16:11:22python-devsetmessages: + msg232433
2014-12-10 08:11:26python-devsetnosy: + python-dev
messages: + msg232415
2014-03-21 08:26:03ronaldoussorensetmessages: + msg214333
2014-03-07 13:48:26piotr.dobrogostsetnosy: + piotr.dobrogost
2014-02-28 13:23:32ronaldoussorensetmessages: + msg212428
2014-02-28 08:28:01ned.deilysetmessages: + msg212414
2014-02-28 08:09:19christian.heimessetmessages: + msg212412
2014-01-31 19:28:40ned.deilysetmessages: + msg209814
versions: + Python 3.5, - Python 3.4
2013-08-24 22:35:48dstufftsetnosy: + dstufft
2013-07-21 18:30:14mlensethgrepos: + hgrepo204
messages: + msg193461
2013-07-08 12:10:28christian.heimessetmessages: + msg192643
2013-07-08 11:54:39mlensetmessages: + msg192636
2013-07-08 11:48:49christian.heimessetmessages: + msg192634
2013-07-08 06:15:25ronaldoussorensetmessages: + msg192617
2013-07-08 00:18:07ned.deilysetmessages: + msg192604
2013-07-07 23:45:03christian.heimessetmessages: + msg192602
2013-07-07 22:43:05mlensetnosy: + mlen
messages: + msg192600
2013-05-14 18:41:05ronaldoussorensetmessages: + msg189244
2013-05-14 09:16:24ronaldoussorensetmessages: + msg189204
2013-05-12 16:06:05dilettantsetnosy: + dilettant
2013-03-05 07:12:58orsenthilsetmessages: + msg183512
2013-02-11 07:59:47ronaldoussorensetmessages: + msg181880
2013-02-10 20:26:04orsenthilsetnosy: + orsenthil
messages: + msg181850
2013-02-08 09:34:07ronaldoussorensetmessages: + msg181664
2013-02-08 08:18:57ned.deilysetpriority: release blocker -> high

messages: + msg181663
versions: - Python 2.7, Python 3.2, Python 3.3
2013-02-05 13:19:57ronaldoussorensetmessages: + msg181435
2013-02-05 13:14:10christian.heimessetmessages: + msg181433
2013-02-05 12:57:05christian.heimessetnosy: + christian.heimes
2013-02-05 06:53:50ronaldoussorensetmessages: + msg181420
2013-02-05 03:18:03ned.deilysetmessages: + msg181410
2013-02-05 03:08:41eric.araujosetnosy: + pitrou, eric.araujo
messages: + msg181408
2013-02-04 22:48:12ned.deilysetnosy: + loewis, brian.curtin
messages: + msg181397
2013-02-04 21:46:09ronaldoussorensetmessages: + msg181392
2013-02-04 21:27:27benjamin.petersonsetmessages: + msg181389
2013-02-04 21:23:30esc24setnosy: + esc24
2013-02-04 18:31:20ned.deilycreate