classification
Title: Support OpenSSL 1.1.1
Type: enhancement Stage: patch review
Components: SSL Versions: Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: alex, benjamin.peterson, christian.heimes, cstratak, dstufft, hroncok, janssen, kroeckx, ned.deily, yan12125
Priority: high Keywords: patch

Created on 2018-02-25 11:54 by christian.heimes, last changed 2018-09-30 11:00 by christian.heimes.

Pull Requests
URL Status Linked Edit
PR 5663 merged christian.heimes, 2018-02-25 12:06
PR 5923 merged miss-islington, 2018-02-27 07:56
PR 8760 merged christian.heimes, 2018-08-14 07:40
PR 8761 merged christian.heimes, 2018-08-14 07:44
Messages (15)
msg312804 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-25 11:54
I'm using this ticket as an epos to track commits and required changes for OpenSSL 1.1.1 and TLS 1.3. Fixes need to be backported to 2.7 and 3.6 to 3.8. We might have to consider backports to 3.4 and 3.5, too.

If all goes to plan, OpenSSL 1.1.1 final is scheduled for 8th May 2018, https://www.openssl.org/policies/releasestrat.html . It will contain support for TLS 1.3. Python should either support TLS 1.3 by then or disable TLS 1.3 by default.

Fixes:

* #20995 added TLS 1.3 cipher suite support
* #29136 added OP_NO_TLSv1_3
* #30622 fixes NPN guard for OpenSSL 1.1.1

Issues:

* A new option OP_ENABLE_MIDDLEBOX_COMPAT is enabled by default. We need to expose the flag to make test pass.
* TLS 1.3 has changed session handling. The current session code cannot handle TLS 1.3 session resumption.
* Threaded echo server and asynchat based tests are failing with TLS 1.3. I haven't analyzed the issue properly. It looks like the server thread dies when a handshake error occurs.
msg312897 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-26 08:52
Ned, Benjamin

OpenSSL 1.1.1 is scheduled to be released just before 3.7.0rc1 will come out. I'd rather address as many issues now instead of adding last minute patches to the release candidate.

Once OpenSSL 1.1.1 is out and Python 3.7 supports it, we can discuss backports. 3.6 and 2.7 are going to need fixing, too.
msg317529 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 05:47
Christian, I'm not sure how this issue now differs from Issue33618 (and whether it can be closed as a duplicate) but, with the delay in OpenSSL 1.1.1 and as discussed over there, full 1.1.1 support will have to wait for 3.7.1 et al so I'm removing the "deferred blocker" status.
msg317575 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-05-24 14:17
3.7 and 3.8 support OpenSSL 1.1.1-pre7-dev. For 3.6 and 2.7 I have to backport some test fixes and documentation.

I prefer to wait until both TLS 1.3 and OpenSSL 1.1.1 have been finalized. Once 1.1.1 is out, I'll fix the outstanding issues on master and then backport all fixes to 3.6 / 2.7.
msg322490 - (view) Author: Miro Hrončok (hroncok) * Date: 2018-07-27 13:23
Note that Fedora 29 updated openssl from 1.1.0h to 1.1.1-0.pre8 and Python 3.7 tests are failing. Not 100% sure it's related, but full report at: 

https://bugzilla.redhat.com/show_bug.cgi?id=1609291
msg322795 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-07-31 16:07
Yes test_poplib and test_ftplib on fedora rawhide when run against openssl 1.1.1 pre8. Haven't tried the pr7, but assuming that the tests were fine before here is the list of changes between pre7 and pre8:

https://github.com/openssl/openssl/compare/OpenSSL_1_1_1-pre7...OpenSSL_1_1_1-pre8
msg323526 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-08-14 14:56
New changeset 2a4ee8aa01d61b6a9c8e9c65c211e61bdb471826 by Christian Heimes in branch '3.6':
bpo-32947: Fixes for TLS 1.3 and OpenSSL 1.1.1 (GH-8761)
https://github.com/python/cpython/commit/2a4ee8aa01d61b6a9c8e9c65c211e61bdb471826
msg323901 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-08-22 18:59
This are the errors I'm currently getting testing with the pre9 verion in Debian:
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python2.7/865936/log.gz
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python3.6/865937/log.gz
https://ci.debian.net/data/autopkgtest/testing/amd64/p/python3.7/865938/log.gz
msg323903 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-08-22 19:07
Kurt, can you try again with a current git checkout from master? I fixed a couple of issues lately. CPython master passes all tests with vanilla OpenSSL 1.1.1-pre9. Does Debian change some default settings?
msg323909 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-08-22 20:20
This are automated tests for the packages in Debian. I uploaded the pre9 version to unstable, and as a result of that all reverse dependencies got tested. I don't have any experience with python myself.

Anyway, the openssl.cnf in Debian contains:
[system_default_sect]
MinProtocol = TLSv1.2
CipherString = DEFAULT@SECLEVEL=2

So you might want to override that during the test suite by calling SSL_CTX_set_min_proto_version(ctx, 0).
msg325793 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-09-19 20:16
Christian,

Do you have any update on this? Any idea when we can expect relased python versions that work with OpenSSL 1.1.1?
msg325795 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-09-19 20:18
Soonish, I'm still working on post handshake auth.
msg325926 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-09-20 19:46
Kurt, see #34670 for PHA for server and client side.
msg326719 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-09-30 10:20
Do you have any idea when the next release will be? I think python is currently our biggest blocker for getting OpenSSL 1.1.1 in Debian testing.
msg326720 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-09-30 11:00
The release candidates came out a couple of days ago.
History
Date User Action Args
2018-09-30 11:00:34christian.heimessetmessages: + msg326720
2018-09-30 10:20:06kroeckxsetmessages: + msg326719
2018-09-20 19:46:32christian.heimessetmessages: + msg325926
2018-09-19 20:18:37christian.heimessetmessages: + msg325795
2018-09-19 20:16:44kroeckxsetmessages: + msg325793
2018-08-22 20:20:38kroeckxsetmessages: + msg323909
2018-08-22 19:07:54christian.heimessetmessages: + msg323903
2018-08-22 18:59:45kroeckxsetnosy: + kroeckx
messages: + msg323901
2018-08-14 14:56:35christian.heimessetmessages: + msg323526
2018-08-14 07:44:21christian.heimessetpull_requests: + pull_request8237
2018-08-14 07:40:21christian.heimessetpull_requests: + pull_request8236
2018-07-31 16:07:56cstrataksetnosy: + cstratak
messages: + msg322795
2018-07-30 04:56:23yan12125setnosy: + yan12125
2018-07-27 13:23:58hroncoksetnosy: + hroncok
messages: + msg322490
2018-05-24 14:17:09christian.heimessetmessages: + msg317575
versions: - Python 3.7, Python 3.8
2018-05-24 05:47:40ned.deilysetpriority: deferred blocker -> high

messages: + msg317529
2018-02-27 07:56:49miss-islingtonsetpull_requests: + pull_request5694
2018-02-26 08:52:09christian.heimessetpriority: normal -> deferred blocker
nosy: + janssen, benjamin.peterson, ned.deily, alex, dstufft
messages: + msg312897

2018-02-25 12:06:34christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request5655
2018-02-25 11:54:40christian.heimescreate