classification
Title: Debian Sid/Buster: Cannot enable TLS 1.0/1.1 with PROTOCOL_TLS
Type: behavior Stage: patch review
Components: SSL Versions: Python 3.6, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Alex Gaynor, adrianv, christian.heimes, doko, kroeckx, miss-islington, ned.deily, njs
Priority: critical Keywords: patch

Created on 2017-09-13 15:48 by adrianv, last changed 2018-05-24 20:36 by kroeckx.

Files
File name Uploaded Description Edit
poc.py adrianv, 2017-09-13 15:48
Pull Requests
URL Status Linked Edit
PR 3662 closed christian.heimes, 2017-09-19 19:09
PR 5259 merged christian.heimes, 2018-02-26 11:40
PR 5926 merged miss-islington, 2018-02-27 10:55
Messages (28)
msg302081 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 15:48
According to the documentation (https://docs.python.org/2/library/ssl.html#ssl.PROTOCOL_TLS), using ssl_version = ssl.PROTOCOL_TLS in a server socket should offer all TLS/SSL versions. However, it only offers TLSv1_2.

I attached a proof of concept.


$ python3 poc.py
3.5.4 (default, Aug 12 2017, 14:08:14)
[GCC 7.1.0]
OpenSSL 1.1.0f  25 May 2017
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:719)
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:719)
b'test\n'

$ python2 poc.py
2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118]
OpenSSL 1.1.0f  25 May 2017
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:661)
[SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:661)
test


To connect with s_client:

 $ for i in {tls1,tls1_1,tls1_2} ; do echo test | openssl s_client -connect localhost:3333 -CAfile server.pem -quiet -$i ; done
140164347663616:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../ssl/record/rec_layer_s3.c:1399:SSL alert number 70
139926441944320:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:../ssl/record/rec_layer_s3.c:1399:SSL alert number 70
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd
verify return:1
read:errno=0
msg302082 - (view) Author: Alex Gaynor (Alex Gaynor) Date: 2017-09-13 15:49
What operating system are you on?
msg302084 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 15:53
Debian buster/sid
msg302085 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 16:03
Debian Buster has patched OpenSSL to disable TLS 1.0 and 1.1 by default, 
https://lists.debian.org/debian-devel-announce/2017/08/msg00004.html
msg302086 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 16:10
I read about that, but I don't understand. If I use openssl s_server -port 3333  , I can connect using either one of the three protocols.

Even if that's the new default, is there no way now to get python on Buster/Sid to use OpenSSL in a non-default mode and have it offer all three versions?
msg302088 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 16:16
You have to enable the protocols by applying a reverse bitmask to SSLContext.options:

ctx = ssl.SSLContext(ssl.PROTOCOL_TLS)
ctx.load_cert_chain('server.pem')
ctx.options &= ~(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)

sslsock = ctx.wrap_socket(s, server_side=True)
msg302091 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 16:33
Doesn't seem to do anything:

>>> ctx.options
2181170175L
>>> ctx.options & ~(ssl.OP_NO_TLSv1 | ssl.OP_NO_TLSv1_1)
2181170175L
msg302092 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 16:37
Please report this issue to the Debian maintainers. I don't know how Debian has disabled TLS 1.0 and TLS 1.1 for the SSL_METHOD *TLS_method(void). It might not be possible to enable auto-negotiation for old protocols at all.
msg302093 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 16:41
Okay, thanks for your time!
msg302094 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 16:41
Ah, here we go: https://anonscm.debian.org/viewvc/pkg-openssl/openssl/branches/1.1.0/debian/patches/tls1_2_default.patch

Debian patched the default for SSL_CTX_set_min_proto_version(). The SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version() API calls are OpenSSL 1.1.0-only and not available from Python. It is not possible to override the minimum version from Python.
msg302097 - (view) Author: Adrian Vollmer (adrianv) Date: 2017-09-13 17:07
I have a workaround for now:

        versions = [ssl.PROTOCOL_TLSv1,
                    ssl.PROTOCOL_TLSv1_1,
                    ssl.PROTOCOL_TLSv1_2,
                   ]
        firstbytes = s.recv(16, socket.MSG_PEEK)
        ss = ssl.wrap_socket(
            s,
            server_side=True,
            certfile="server.pem",
            keyfile="server.pem",
            #  ssl_version=versions[ord(firstbytes[10])-1] # python2
            ssl_version=versions[firstbytes[10]-1]
        )

How much of an ugly hack is this? :)
msg302099 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 17:10
It's an ugly hack and not a long term solution. The PROTOCOL_TLSv* constants and ssl.wrap_socket() are discouraged and will be removed soon.
msg302100 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-13 17:11
Matthias, this issue affects Debian and probably Ubuntu, too. Could you please discuss it with Debian maintainers and propose a workaround? Python does not expose the new OpenSSL 1.1.0 SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version() calls. We only support SSL_CTX_set_options() with SSL_OP_NO_TLSv1 and SSL_OP_NO_TLSv1_1.
msg302561 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-19 19:19
PR 3662 undos Debian's patching of OpenSSL. I'm not keen to undo a security improvement. However Debian is breaking backwards compatibility. For Python 3.7 we could consider to disable TLS 1.0 and TLS 1.1 for PROTOCOL_TLS_SERVER and PROTOCOL_TLS_CLIENT.
msg307105 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-11-28 01:47
FWIW, Debian seems to have re-enabled TLS 1.0 and 1.1 in "testing".  As a result, test_ssl now passes again.

openssl (1.1.0g-1) unstable; urgency=medium

  * New upstream version
    - Fixes CVE-2017-3735
    - Fixes CVE-2017-3736
  * Remove patches applied upstream
  * Temporary enable TLS 1.0 and 1.1 again (#875423)
  * Attempt to fix testsuite race condition
  * update no-symbolic.patch to apply

 -- Kurt Roeckx <kurt@roeckx.be>  Thu, 02 Nov 2017 15:22:48 +0100
msg310996 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-28 21:11
I'm not clear if this is still needed, i.e. has Debian backed off on their change across the board?  If it is still needed, I'm going to allow an extension for landing of it until 3.7.0b2, currently scheduled for 2018-02-26.  If anyone else can help Christian get this in before b2, that would be great.  I'm removing older versions for now. We can discuss potential backports after the code lands.
msg311008 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-01-28 21:51
The feature is still useful -- whether or not Debian disables TLS 1.0 and 1.1. The new API is easier to use and more convenient than the old bitmask approach. "option &=~OP_NO_SSLv3" is just horrible. :)
msg312920 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-26 11:48
I have closed the feature newer BPO-32609 in favor of this bug because Ned gave this bug a deferred blocker priority.


OpenSSL 1.1 has introduced a new API to set the minimum and maximum supported protocol version. The API is easier to use than the old OP_NO_TLSv1 option flags, too

https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_min_proto_version.html

Debian used the new setters to disable TLS 1.0 and 1.1 in testing, #31453. The old TLS versions have been enabled again for now. Python must expose the new API in case Debian decides to disable them again. Another $DIST has considered to implement a virtually the same policy as Debian.

I also like to deprecate the old OP_NO_TLSv1 et al. flags in favor of the new API. The option flags are awkward to use and easy to get wrong. For example applications must not leave holes in the OP_NO range (e.g. allow TLS 1.0 and 1.2 but disable 1.1).
msg312991 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-27 10:01
I have another good reason to land PR 5259 in 3.7. OpenSSL 1.1.0 has deprecated the old way to disable/enable protocol versions with set option. The OP_NO_TLSv1* constants will likely get removed in OpenSSL 1.2.0. I'm expecting to see a 1.2.0 release within the next two years. Therefore we should include SSLContext.maximum_version and SSLContext.minimum_version now. We may even have to backport them to 3.6 and 2.7.

https://www.openssl.org/docs/man1.1.0/ssl/SSL_CTX_set_options.html

> SSL_OP_NO_SSLv3, SSL_OP_NO_TLSv1, SSL_OP_NO_TLSv1_1, SSL_OP_NO_TLSv1_2, SSL_OP_NO_DTLSv1, SSL_OP_NO_DTLSv1_2
>    These options turn off the SSLv3, TLSv1, TLSv1.1 or TLSv1.2 protocol versions with TLS or the DTLSv1, DTLSv1.2 versions with DTLS, respectively. As of OpenSSL 1.1.0, these options are deprecated, use SSL_CTX_set_min_proto_version and SSL_CTX_set_max_proto_version instead.
msg312997 - (view) Author: miss-islington (miss-islington) Date: 2018-02-27 11:41
New changeset 4c842b09209ccf1b4f853106b1f58bb888da02ef by Miss Islington (bot) in branch '3.7':
bpo-31453: Add setter for min/max protocol version (GH-5259)
https://github.com/python/cpython/commit/4c842b09209ccf1b4f853106b1f58bb888da02ef
msg312998 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-02-27 11:47
Ned,

I have pushed the enhancement now. I feel a bit guilty about the last minute push, but I honestly believe it's in our best interesting.

There is a high chance that the new APIs will be required for Debian and other distros in the near future. OpenSSL has deprecated the old API, too. It is likely that OpenSSL 1.2.0 will remove both the options (OP_NO_TLSv1, OP_NO_TLSv1_2, ...) options and version specific protocols (PROTOCOL_TLSv1, PROTOCOL_TLSv1_2, ...).
msg314635 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-03-29 08:05
Christian, is there more needed for this issue or can it be closed?
msg316017 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-01 20:49
Christian, ping.  Can we close this?
msg317534 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-05-24 06:12
Christian, ping again: can this be closed?  In any case, it would not seem to be a "deferred blocker"; downgrading to "critical".  Please close or update, thanks!
msg317574 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-05-24 14:14
The problem no longer affects Python 3.7 and 3.8. It may affects Python 3.6 and 2.7 if Debian to decide to disable TLS 1.0 and 1.1 again. If Debian uses the new OpenSSL 1.1.0 API to disable the protocols, then I have to backport https://bugs.python.org/issue32609 to 2.7 and 3.6.
msg317610 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-05-24 20:09
Note that the version in experimental only supports TLS 1.2 and 1.3 with the default config. It's moved from fixed in the code, to the default config file. I expect to upload that to unstable "soon", at which point people will be affected by this again.
msg317618 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-05-24 20:31
Thanks Kurt,

which API are you using to disable TLS 1.0 and 1.1? Is it the old SSL_CTX_set_options() or the new SSL_CTX_set_min/max_proto_version() API?
msg317619 - (view) Author: Kurt Roeckx (kroeckx) * Date: 2018-05-24 20:36
The effect is the same as calling SSL_CTX_set_min_proto_version().
History
Date User Action Args
2018-05-24 20:36:41kroeckxsetmessages: + msg317619
2018-05-24 20:31:36christian.heimessetmessages: + msg317618
2018-05-24 20:09:45kroeckxsetnosy: + kroeckx
messages: + msg317610
2018-05-24 14:14:46christian.heimessetmessages: + msg317574
2018-05-24 06:12:33ned.deilysetpriority: deferred blocker -> critical

messages: + msg317534
2018-05-15 20:26:48ned.deilysetversions: + Python 2.7, Python 3.6, - Python 3.7, Python 3.8
2018-05-01 20:49:17ned.deilysetmessages: + msg316017
2018-03-29 08:05:10ned.deilysetmessages: + msg314635
2018-02-27 11:47:10christian.heimessetmessages: + msg312998
2018-02-27 11:41:07miss-islingtonsetnosy: + miss-islington
messages: + msg312997
2018-02-27 10:55:52miss-islingtonsetpull_requests: + pull_request5697
2018-02-27 10:01:00christian.heimessetmessages: + msg312991
2018-02-26 11:48:19christian.heimessetnosy: + njs
messages: + msg312920
2018-02-26 11:43:04christian.heimeslinkissue32609 superseder
2018-02-26 11:40:52christian.heimessetpull_requests: + pull_request5675
2018-01-28 21:51:03christian.heimessetmessages: + msg311008
2018-01-28 21:11:30ned.deilysetpriority: normal -> deferred blocker

messages: + msg310996
versions: + Python 3.8, - Python 2.7, Python 3.6
2017-11-28 01:47:09ned.deilysetnosy: + ned.deily
messages: + msg307105
2017-09-19 19:19:36christian.heimessetmessages: + msg302561
2017-09-19 19:09:33christian.heimessetkeywords: + patch
stage: patch review
pull_requests: + pull_request3651
2017-09-13 17:11:20christian.heimessetversions: + Python 3.6, Python 3.7, - Python 3.5
nosy: + doko

messages: + msg302100

assignee: christian.heimes ->
2017-09-13 17:10:59christian.heimessetmessages: + msg302099
2017-09-13 17:07:03adrianvsetmessages: + msg302097
versions: - Python 3.6, Python 3.7
2017-09-13 16:47:43christian.heimessetversions: + Python 3.6, Python 3.7
2017-09-13 16:47:34christian.heimessettitle: ssl.PROTOCOL_TLS only select TLSv1.2 -> Debian Sid/Buster: Cannot enable TLS 1.0/1.1 with PROTOCOL_TLS
2017-09-13 16:41:31christian.heimessetmessages: + msg302094
2017-09-13 16:41:27adrianvsetmessages: + msg302093
2017-09-13 16:37:54christian.heimessetmessages: + msg302092
2017-09-13 16:33:03adrianvsetmessages: + msg302091
2017-09-13 16:16:33christian.heimessetmessages: + msg302088
2017-09-13 16:10:07adrianvsetmessages: + msg302086
2017-09-13 16:03:42christian.heimessetmessages: + msg302085
2017-09-13 15:53:55adrianvsetmessages: + msg302084
2017-09-13 15:49:39Alex Gaynorsetnosy: + Alex Gaynor
messages: + msg302082
2017-09-13 15:48:25adrianvcreate