classification
Title: Make OpenSSL module compatible with OpenSSL 1.1.0
Type: security Stage: resolved
Components: Versions: Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Chi Hsuan Yen, Michael.Felt, alex, christian.heimes, dstufft, giampaolo.rodola, janssen, matrixise, pitrou, python-dev, smpeepers, spil, zach.ware
Priority: normal Keywords: patch

Created on 2016-03-02 12:14 by christian.heimes, last changed 2016-09-08 14:19 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
patch-lang_python35-OpenSSL-1.1.0+LibreSSL spil, 2016-04-16 10:39 Revised patch for OpenSSL 1.1.0 support
Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch christian.heimes, 2016-08-26 13:16 Patch for 2.7
Port-Python-s-SSL-module-to-OpenSSL-1.1.0-3.patch christian.heimes, 2016-08-26 13:17 review
Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-4.patch christian.heimes, 2016-08-31 14:03
Port-Python-s-SSL-module-to-OpenSSL-1.1.0-4.patch christian.heimes, 2016-08-31 14:03 review
Port-Python-s-SSL-module-to-OpenSSL-1.1.0-5.patch christian.heimes, 2016-09-04 15:15 review
Messages (27)
msg261108 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-03-02 12:14
OpenSSL 1.1.0 is changing a lot of internals. Most structs are opaque, RC4 and SSLv2 are gone. I've a rough patch in my private repos. I'll submit the patch as soon as the code is compatible with OpenSSL 1.0.2, too.

https://github.com/python/cpython/compare/master...tiran:feature/openssl110
msg261140 - (view) Author: Michael Felt (Michael.Felt) * Date: 2016-03-02 22:48
Since you are looking, maybe look at whether it is also libreSSL compatible?
msg261906 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-03-17 11:08
Here is a first working patch. It requires 1.1.0-pre4. The failing ALPN test is caused by a regression in OpenSSL.
msg263545 - (view) Author: Bernard Spil (spil) * Date: 2016-04-16 09:37
Testing this patch on HardenedBSD/LibreSSL (base SSL libs replaced with LibreSSL)
msg263546 - (view) Author: Bernard Spil (spil) * Date: 2016-04-16 10:39
Checking version numbers to see if a feature is available is a bad practice. How can features ever be removed this way! Would be better to check for the feature itself (using autoconf).

The patch was mostly OK but any check for OPENSSL_VERSION_NUMBER for now also requires a negative check for LIBRESSL_VERSION_NUMBER as LibreSSL froze features at 1.0.1g.

Next to that, anything requiring compression (CRIME attack) should be guarded using and #infdef OPENSSL_NO_COMP.

This patch allowed me to build Python 3.5 with LibreSSL 2.3 (i.e. without SSLv3, Compression, RC4, SHA-0, etc)
msg263780 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-04-19 22:21
The patch makes Python compatible with OpenSSL 1.1.0-pre6-dev from
git. The ssl and hashlib module are also compatible with OpenSSL 0.9.8zh,
1.0.1s, 1.0.2g as well as LibreSSL 2.3.3.
msg263781 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-04-19 22:23
PS: The patch depends on https://github.com/openssl/openssl/pull/979
msg269851 - (view) Author: Bernard Spil (spil) * Date: 2016-07-05 20:25
Can you please replace the HAVE_RAND_EGD bits with OPENSSL_NO_EGD as defined by both OpenSSL 1.1 and LibreSSL?

EGD default disabled https://github.com/openssl/openssl/blob/master/Configure#L363
EGD methods not available https://github.com/openssl/openssl/blob/master/include/openssl/rand.h#L61
msg272128 - (view) Author: St├ęphane Wirtel (matrixise) * Date: 2016-08-07 18:15
Hi Christian,

I have reviewed your patch, seems to be fine for me.
msg272165 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-08 12:47
St├ęphane, I have addressed your code review.

def __new__() no longer hard-codes protocol. We can change that in a later version of Python. OpenSSL has deprecated all SSL methods except of the generic TLS method. The TLS method was formerly known as SSLv23 method and does auto-negotiation of the latest supported method.

Lib/test/test_ssl.py:1183: LibreSSL does not support SSL_CA_PATH and SSL_CA_DIR env vars. I have changed the comment on the test.

Modules/_hashopenssl.c:127: _hashopenssl.c now does error checks on EVP digest copy. The copy operation can fail when an EVP ENGINE is involved.

HAS_FAST_PKCS5_PBKDF2_HMAC is defined in _hashopenssl.c. OpenSSL used to have a bad implementation of PKBDF2. I fixed it in 2013. The workaround is no longer required for OpenSSL >= 1.1.0. You can find more details in https://jbp.io/2015/08/11/pbkdf2-performance-matters/
msg273675 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-25 19:35
OpenSSL 1.1.0 final was released a couple of hours ago. One test is failing because it uses 3DES. 1.1.0 has 3DES disabled by default.
msg273706 - (view) Author: Chi Hsuan Yen (Chi Hsuan Yen) * Date: 2016-08-26 14:18
There are still quite a few references to `PROTOCOL_SSLv23` in Doc/library/ssl.rst. Should they be updated as well?
msg273723 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2016-08-26 19:41
- The 2.7 patch contains numerous references to 3.6, these should be rewritten to 2.7.x
-
msg273837 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-28 19:22
Thanks Alex, I ported the Python and C code to 2.7 but forgot to address doc updates. You can find an updated patch on github: https://github.com/python/cpython/compare/2.7...tiran:feature/openssl110_27 I'll submit a new patch after your review.

Chi Hsuan Yen, I'll update remaining documentation later.
msg273880 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-08-29 19:28
This will require significant updates to PCbuild/prepare_ssl.py and/or the way we build OpenSSL on Windows before we can even properly test this on Windows.  I don't think that should hold up acceptance of the rest of the patch (provided 1.0.2 support remains intact), but will need to be handled eventually.

Building on Windows with 1.0.2h is broken with the current patch, but I don't understand things well enough to diagnose it:

ssleay.lib(ssl_lib.obj) : error LNK2005: _SSL_CTX_set_default_passwd_cb_userdata already defined in _ssl.obj [P:\ath\to\cpython\PCbuild\_ssl.vcxproj]                                                                                                                                                          
ssleay.lib(ssl_lib.obj) : error LNK2005: _SSL_CTX_set_default_passwd_cb already defined in _ssl.obj [P:\ath\to\cpython\PCbuild\_ssl.vcxproj]
msg273882 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-29 20:39
Hi Zachary, you have found a bug in my patch. I mistakenly defined SSL_CTX_set_default_passwd_cb() and SSL_CTX_set_default_passwd_cb_userdata() for OpenSSL < 1.1.0. Both functions already exist. Only the getters are missing for < 1.1.0. Please remove both functions from _ssl.c and try again.
msg273883 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2016-08-29 20:46
Looks like that took care of it, build succeeded with no new warnings, and test.ssltests passed.
msg273885 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-08-29 20:57
Awesome! I have removed the surplus functions, made the other additional functions static and fixed minor test issue with LibreSSL and OpenSSL < 1.0.1. My branches on github compile and pass all tests with OpenSSL "0.9.8zc", "0.9.8zh", "1.0.1t", "1.0.2", "1.0.2h", "1.1.0" and LibreSSL "2.3.0", "2.4.2" on Linux X86_86. I'm using my script https://github.com/tiran/multissl for testing.
msg274110 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2016-09-01 09:28
Christian, thanks a lot for doing this!  Do you plan to change the SSLContext constructor and make the protocol argument optional? It sounds like that would be a logical followup to the OpenSSL API changes.
msg274162 - (view) Author: Bernard Spil (spil) * Date: 2016-09-01 18:19
Hi Christian,

Great stuff!

Please can you replace the HAVE_RAND_EGD ifdefs into OPENSSL_NO_EGD checks? Then the RAND_egd checks in configure.ac can also be removed.

This was introduced by OpenSSL in https://github.com/openssl/openssl/commit/0423f812dc61f70c6ae6643191259ca9e5692c7f and is consistent with the naming in LibreSSL.

Cheers,

Bernard.
msg274219 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-02 09:14
Bernard, where do you see HAVE_RAND_EGD in my patch or in any recent version of _ssl.c? There is no reference to HAVE_RAND_EGD. The patches use OPENSSL_NO_EGD.
msg274220 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-02 09:25
Yes, I plan to change this for Python 3.7 along with #27876.
msg274224 - (view) Author: Bernard Spil (spil) * Date: 2016-09-02 10:37
Sorry for the noise Christian, I thought the former EGD handling was still in place. That was fixed with https://github.com/python/cpython/commit/968ec1d29b44ca7a600df5984adff00a78392368 on 07 Jul 2016
msg274359 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-09-04 11:51
Antoine, I have reconsidered your idea. Let's make the default value PROTOCOL_TLS in 3.6 and deprecated the other protocol methods. We can remove them in 3.8 or 3.9. I'll push another patch later today.
msg274438 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-05 21:38
New changeset 5c75b315152b by Christian Heimes in branch '3.5':
Issue #26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/5c75b315152b

New changeset bc5ba11973f5 by Christian Heimes in branch 'default':
Issue #26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/bc5ba11973f5

New changeset 14b611ddaabe by Christian Heimes in branch '2.7':
Issue #26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/14b611ddaabe
msg274460 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-05 23:13
New changeset 5566732c8ac5 by Christian Heimes in branch '3.5':
Issue #26470: Use short name rather than name for compression name to fix #27958.
https://hg.python.org/cpython/rev/5566732c8ac5

New changeset 2593ed9a6a62 by Christian Heimes in branch '2.7':
Issue #26470: Use short name rather than name for compression name to fix #27958.
https://hg.python.org/cpython/rev/2593ed9a6a62
msg274462 - (view) Author: Roundup Robot (python-dev) Date: 2016-09-05 23:14
New changeset d92f26a53b70 by Christian Heimes in branch 'default':
Issue #26470: Use short name rather than name for compression name to fix #27958.
https://hg.python.org/cpython/rev/d92f26a53b70
History
Date User Action Args
2016-09-08 14:19:46christian.heimessetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2016-09-06 11:16:51christian.heimesunlinkissue27866 dependencies
2016-09-05 23:14:46python-devsetmessages: + msg274462
2016-09-05 23:13:56python-devsetmessages: + msg274460
2016-09-05 21:38:15python-devsetnosy: + python-dev
messages: + msg274438
2016-09-04 15:15:10christian.heimessetfiles: + Port-Python-s-SSL-module-to-OpenSSL-1.1.0-5.patch
2016-09-04 11:51:02christian.heimessetmessages: + msg274359
2016-09-02 10:37:33spilsetmessages: + msg274224
2016-09-02 09:25:58christian.heimessetmessages: + msg274220
2016-09-02 09:14:33christian.heimessetmessages: + msg274219
2016-09-01 18:19:19spilsetmessages: + msg274162
2016-09-01 09:28:34pitrousetmessages: + msg274110
2016-08-31 20:10:58christian.heimeslinkissue27866 dependencies
2016-08-31 14:03:55christian.heimessetfiles: + Port-Python-s-SSL-module-to-OpenSSL-1.1.0-4.patch
2016-08-31 14:03:41christian.heimessetfiles: + Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-4.patch
2016-08-29 20:57:07christian.heimessetmessages: + msg273885
2016-08-29 20:46:36zach.waresetmessages: + msg273883
2016-08-29 20:39:03christian.heimessetmessages: + msg273882
2016-08-29 19:28:04zach.waresetmessages: + msg273880
2016-08-28 19:22:44christian.heimessetmessages: + msg273837
2016-08-26 19:41:30alexsetmessages: + msg273723
2016-08-26 14:18:41Chi Hsuan Yensetmessages: + msg273706
2016-08-26 13:17:27christian.heimessetfiles: + Port-Python-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-26 13:17:04christian.heimessetfiles: - Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-26 13:16:53christian.heimessetfiles: - Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-26 13:16:41christian.heimessetfiles: - Port-Python-s-SSL-module-to-OpenSSL-1.1.0-2.patch
2016-08-26 13:16:32christian.heimessetfiles: - Port-Python-s-SSL-module-to-OpenSSL-1.1.0.patch
2016-08-26 13:16:24christian.heimessetfiles: - Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0.patch
2016-08-26 13:16:10christian.heimessetfiles: + Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-26 13:15:48christian.heimessetfiles: + Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-26 13:15:29christian.heimessetfiles: + Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch
2016-08-25 19:35:04christian.heimessetmessages: + msg273675
2016-08-25 19:30:07Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2016-08-15 13:55:33christian.heimeslinkissue27766 dependencies
2016-08-08 12:47:12christian.heimessetfiles: + Port-Python-s-SSL-module-to-OpenSSL-1.1.0-2.patch

messages: + msg272165
2016-08-07 18:15:35matrixisesetnosy: + matrixise
messages: + msg272128
2016-08-07 17:55:29christian.heimessetfiles: + Port-Python-s-SSL-module-to-OpenSSL-1.1.0.patch
2016-08-07 17:55:15christian.heimessetfiles: + Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0.patch
2016-08-07 17:54:51christian.heimessetfiles: - 0001-Port-Python-s-SSL-module-to-OpenSSL-1.1.0-WIP.patch
2016-07-05 20:25:57spilsetmessages: + msg269851
2016-06-12 11:22:24christian.heimessetassignee: christian.heimes ->
2016-04-19 22:23:17christian.heimessetmessages: + msg263781
2016-04-19 22:21:53christian.heimessetfiles: + 0001-Port-Python-s-SSL-module-to-OpenSSL-1.1.0-WIP.patch

messages: + msg263780
2016-04-19 22:21:22christian.heimessetfiles: - 0001-Port-Python-s-SSL-module-to-OpenSSL-1.1.0-WIP.patch
2016-04-16 10:39:35spilsetfiles: + patch-lang_python35-OpenSSL-1.1.0+LibreSSL

messages: + msg263546
2016-04-16 09:37:34spilsetnosy: + spil
messages: + msg263545
2016-03-17 11:15:02vstinnersetnosy: - vstinner
2016-03-17 11:08:20christian.heimessetfiles: + 0001-Port-Python-s-SSL-module-to-OpenSSL-1.1.0-WIP.patch
versions: + Python 3.6
messages: + msg261906

keywords: + patch
stage: needs patch -> patch review
2016-03-08 11:58:26vstinnersetnosy: + vstinner
2016-03-07 17:37:59smpeeperssetnosy: + smpeepers
2016-03-02 22:48:45Michael.Feltsetnosy: + Michael.Felt
messages: + msg261140
2016-03-02 14:33:16zach.waresetnosy: + zach.ware
2016-03-02 12:14:53christian.heimescreate