Make OpenSSL module compatible with OpenSSL 1.1.0 #70657

tiran · 2016-03-02T12:14:54Z

BPO	26470
Nosy	@pitrou, @giampaolo, @tiran, @alex, @zware, @dstufft, @matrixise, @aixtools, @yan12125
PRs	[3.4] bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0. #12211 [3.5] bpo-36576: Skip test_ssl and test_asyncio tests failing with OpenSSL 1.1.1 #12694
Files	patch-lang_python35-OpenSSL-1.1.0+LibreSSL: Revised patch for OpenSSL 1.1.0 support Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-3.patch: Patch for 2.7 Port-Python-s-SSL-module-to-OpenSSL-1.1.0-3.patch Port-Python-2.7-s-SSL-module-to-OpenSSL-1.1.0-4.patch Port-Python-s-SSL-module-to-OpenSSL-1.1.0-4.patch Port-Python-s-SSL-module-to-OpenSSL-1.1.0-5.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-08.14:19:46.658>
created_at = <Date 2016-03-02.12:14:53.739>
labels = ['type-security']
title = 'Make OpenSSL module compatible with OpenSSL 1.1.0'
updated_at = <Date 2019-04-05.08:27:39.831>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2019-04-05.08:27:39.831>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-08.14:19:46.658>
closer = 'christian.heimes'
components = []
creation = <Date 2016-03-02.12:14:53.739>
creator = 'christian.heimes'
dependencies = []
files = ['42480', '44231', '44232', '44296', '44297', '44360']
hgrepos = []
issue_num = 26470
keywords = ['patch']
message_count = 27.0
messages = ['261108', '261140', '261906', '263545', '263546', '263780', '263781', '269851', '272128', '272165', '273675', '273706', '273723', '273837', '273880', '273882', '273883', '273885', '274110', '274162', '274219', '274220', '274224', '274359', '274438', '274460', '274462']
nosy_count = 13.0
nosy_names = ['janssen', 'pitrou', 'giampaolo.rodola', 'christian.heimes', 'alex', 'python-dev', 'zach.ware', 'dstufft', 'matrixise', 'spil', 'Michael.Felt', 'yan12125', 'smpeepers']
pr_nums = ['12211', '12694']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue26470'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6']

tiran · 2016-03-02T12:14:54Z

OpenSSL 1.1.0 is changing a lot of internals. Most structs are opaque, RC4 and SSLv2 are gone. I've a rough patch in my private repos. I'll submit the patch as soon as the code is compatible with OpenSSL 1.0.2, too.

master...tiran:feature/openssl110

aixtools · 2016-03-02T22:48:45Z

Since you are looking, maybe look at whether it is also libreSSL compatible?

tiran · 2016-03-17T11:08:18Z

Here is a first working patch. It requires 1.1.0-pre4. The failing ALPN test is caused by a regression in OpenSSL.

spil · 2016-04-16T09:37:35Z

Testing this patch on HardenedBSD/LibreSSL (base SSL libs replaced with LibreSSL)

spil · 2016-04-16T10:39:28Z

Checking version numbers to see if a feature is available is a bad practice. How can features ever be removed this way! Would be better to check for the feature itself (using autoconf).

The patch was mostly OK but any check for OPENSSL_VERSION_NUMBER for now also requires a negative check for LIBRESSL_VERSION_NUMBER as LibreSSL froze features at 1.0.1g.

Next to that, anything requiring compression (CRIME attack) should be guarded using and #infdef OPENSSL_NO_COMP.

This patch allowed me to build Python 3.5 with LibreSSL 2.3 (i.e. without SSLv3, Compression, RC4, SHA-0, etc)

tiran · 2016-04-19T22:21:53Z

The patch makes Python compatible with OpenSSL 1.1.0-pre6-dev from
git. The ssl and hashlib module are also compatible with OpenSSL 0.9.8zh,
1.0.1s, 1.0.2g as well as LibreSSL 2.3.3.

tiran · 2016-04-19T22:23:17Z

PS: The patch depends on openssl/openssl#979

spil · 2016-07-05T20:25:57Z

Can you please replace the HAVE_RAND_EGD bits with OPENSSL_NO_EGD as defined by both OpenSSL 1.1 and LibreSSL?

EGD default disabled https://github.com/openssl/openssl/blob/master/Configure#L363
EGD methods not available https://github.com/openssl/openssl/blob/master/include/openssl/rand.h#L61

matrixise · 2016-08-07T18:15:35Z

Hi Christian,

I have reviewed your patch, seems to be fine for me.

tiran · 2016-08-08T12:47:11Z

Stéphane, I have addressed your code review.

def __new__() no longer hard-codes protocol. We can change that in a later version of Python. OpenSSL has deprecated all SSL methods except of the generic TLS method. The TLS method was formerly known as SSLv23 method and does auto-negotiation of the latest supported method.

Lib/test/test_ssl.py:1183: LibreSSL does not support SSL_CA_PATH and SSL_CA_DIR env vars. I have changed the comment on the test.

Modules/_hashopenssl.c:127: _hashopenssl.c now does error checks on EVP digest copy. The copy operation can fail when an EVP ENGINE is involved.

HAS_FAST_PKCS5_PBKDF2_HMAC is defined in _hashopenssl.c. OpenSSL used to have a bad implementation of PKBDF2. I fixed it in 2013. The workaround is no longer required for OpenSSL >= 1.1.0. You can find more details in https://jbp.io/2015/08/11/pbkdf2-performance-matters/

tiran · 2016-08-25T19:35:05Z

OpenSSL 1.1.0 final was released a couple of hours ago. One test is failing because it uses 3DES. 1.1.0 has 3DES disabled by default.

yan12125 · 2016-08-26T14:18:41Z

There are still quite a few references to PROTOCOL_SSLv23 in Doc/library/ssl.rst. Should they be updated as well?

alex · 2016-08-26T19:41:31Z

The 2.7 patch contains numerous references to 3.6, these should be rewritten to 2.7.x

tiran · 2016-08-28T19:22:45Z

Thanks Alex, I ported the Python and C code to 2.7 but forgot to address doc updates. You can find an updated patch on github: 2.7...tiran:feature/openssl110_27 I'll submit a new patch after your review.

Chi Hsuan Yen, I'll update remaining documentation later.

zware · 2016-08-29T19:28:04Z

This will require significant updates to PCbuild/prepare_ssl.py and/or the way we build OpenSSL on Windows before we can even properly test this on Windows. I don't think that should hold up acceptance of the rest of the patch (provided 1.0.2 support remains intact), but will need to be handled eventually.

Building on Windows with 1.0.2h is broken with the current patch, but I don't understand things well enough to diagnose it:

ssleay.lib(ssl_lib.obj) : error LNK2005: _SSL_CTX_set_default_passwd_cb_userdata already defined in _ssl.obj [P:\ath\to\cpython\PCbuild\_ssl.vcxproj]
ssleay.lib(ssl_lib.obj) : error LNK2005: _SSL_CTX_set_default_passwd_cb already defined in _ssl.obj [P:\ath\to\cpython\PCbuild\_ssl.vcxproj]

tiran · 2016-08-29T20:39:03Z

Hi Zachary, you have found a bug in my patch. I mistakenly defined SSL_CTX_set_default_passwd_cb() and SSL_CTX_set_default_passwd_cb_userdata() for OpenSSL < 1.1.0. Both functions already exist. Only the getters are missing for < 1.1.0. Please remove both functions from _ssl.c and try again.

zware · 2016-08-29T20:46:36Z

Looks like that took care of it, build succeeded with no new warnings, and test.ssltests passed.

tiran · 2016-08-29T20:57:08Z

Awesome! I have removed the surplus functions, made the other additional functions static and fixed minor test issue with LibreSSL and OpenSSL < 1.0.1. My branches on github compile and pass all tests with OpenSSL "0.9.8zc", "0.9.8zh", "1.0.1t", "1.0.2", "1.0.2h", "1.1.0" and LibreSSL "2.3.0", "2.4.2" on Linux X86_86. I'm using my script https://github.com/tiran/multissl for testing.

pitrou · 2016-09-01T09:28:34Z

Christian, thanks a lot for doing this! Do you plan to change the SSLContext constructor and make the protocol argument optional? It sounds like that would be a logical followup to the OpenSSL API changes.

spil · 2016-09-01T18:19:19Z

Hi Christian,

Great stuff!

Please can you replace the HAVE_RAND_EGD ifdefs into OPENSSL_NO_EGD checks? Then the RAND_egd checks in configure.ac can also be removed.

This was introduced by OpenSSL in openssl/openssl@0423f81 and is consistent with the naming in LibreSSL.

Cheers,

Bernard.

tiran · 2016-09-02T09:14:33Z

Bernard, where do you see HAVE_RAND_EGD in my patch or in any recent version of _ssl.c? There is no reference to HAVE_RAND_EGD. The patches use OPENSSL_NO_EGD.

tiran · 2016-09-02T09:25:58Z

Yes, I plan to change this for Python 3.7 along with bpo-27876.

spil · 2016-09-02T10:37:33Z

Sorry for the noise Christian, I thought the former EGD handling was still in place. That was fixed with 968ec1d on 07 Jul 2016

tiran · 2016-09-04T11:51:03Z

Antoine, I have reconsidered your idea. Let's make the default value PROTOCOL_TLS in 3.6 and deprecated the other protocol methods. We can remove them in 3.8 or 3.9. I'll push another patch later today.

python-dev · 2016-09-05T21:38:15Z

New changeset 5c75b315152b by Christian Heimes in branch '3.5':
Issue bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/5c75b315152b

New changeset bc5ba11973f5 by Christian Heimes in branch 'default':
Issue bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/bc5ba11973f5

New changeset 14b611ddaabe by Christian Heimes in branch '2.7':
Issue bpo-26470: Port ssl and hashlib module to OpenSSL 1.1.0.
https://hg.python.org/cpython/rev/14b611ddaabe

python-dev · 2016-09-05T23:13:57Z

New changeset 5566732c8ac5 by Christian Heimes in branch '3.5':
Issue bpo-26470: Use short name rather than name for compression name to fix bpo-27958.
https://hg.python.org/cpython/rev/5566732c8ac5

New changeset 2593ed9a6a62 by Christian Heimes in branch '2.7':
Issue bpo-26470: Use short name rather than name for compression name to fix bpo-27958.
https://hg.python.org/cpython/rev/2593ed9a6a62

python-dev · 2016-09-05T23:14:46Z

New changeset d92f26a53b70 by Christian Heimes in branch 'default':
Issue bpo-26470: Use short name rather than name for compression name to fix bpo-27958.
https://hg.python.org/cpython/rev/d92f26a53b70

tiran self-assigned this Mar 2, 2016

tiran added the type-security A security issue label Mar 2, 2016

tiran removed their assignment Jun 12, 2016

tiran closed this as completed Sep 8, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make OpenSSL module compatible with OpenSSL 1.1.0 #70657

Make OpenSSL module compatible with OpenSSL 1.1.0 #70657

tiran commented Mar 2, 2016

tiran commented Mar 2, 2016

aixtools commented Mar 2, 2016

tiran commented Mar 17, 2016

spil mannequin commented Apr 16, 2016

spil mannequin commented Apr 16, 2016

tiran commented Apr 19, 2016

tiran commented Apr 19, 2016

spil mannequin commented Jul 5, 2016

matrixise commented Aug 7, 2016

tiran commented Aug 8, 2016

tiran commented Aug 25, 2016

yan12125 mannequin commented Aug 26, 2016

alex commented Aug 26, 2016

tiran commented Aug 28, 2016

zware commented Aug 29, 2016

tiran commented Aug 29, 2016

zware commented Aug 29, 2016

tiran commented Aug 29, 2016

pitrou commented Sep 1, 2016

spil mannequin commented Sep 1, 2016

tiran commented Sep 2, 2016

tiran commented Sep 2, 2016

spil mannequin commented Sep 2, 2016

tiran commented Sep 4, 2016

python-dev mannequin commented Sep 5, 2016

python-dev mannequin commented Sep 5, 2016

python-dev mannequin commented Sep 5, 2016

Make OpenSSL module compatible with OpenSSL 1.1.0 #70657

Make OpenSSL module compatible with OpenSSL 1.1.0 #70657

Comments

tiran commented Mar 2, 2016

tiran commented Mar 2, 2016

aixtools commented Mar 2, 2016

tiran commented Mar 17, 2016

spil mannequin commented Apr 16, 2016

spil mannequin commented Apr 16, 2016

tiran commented Apr 19, 2016

tiran commented Apr 19, 2016

spil mannequin commented Jul 5, 2016

matrixise commented Aug 7, 2016

tiran commented Aug 8, 2016

tiran commented Aug 25, 2016

yan12125 mannequin commented Aug 26, 2016

alex commented Aug 26, 2016

tiran commented Aug 28, 2016

zware commented Aug 29, 2016

tiran commented Aug 29, 2016

zware commented Aug 29, 2016

tiran commented Aug 29, 2016

pitrou commented Sep 1, 2016

spil mannequin commented Sep 1, 2016

tiran commented Sep 2, 2016

tiran commented Sep 2, 2016

spil mannequin commented Sep 2, 2016

tiran commented Sep 4, 2016

python-dev mannequin commented Sep 5, 2016

python-dev mannequin commented Sep 5, 2016

python-dev mannequin commented Sep 5, 2016