classification
Title: test_ssl fails in Ubuntu 20.04: test_min_max_version_mismatch
Type: compile error Stage: patch review
Components: Build, SSL, Tests Versions: Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Vladyslav.Bondar, bugsrep, christian.heimes, skip.montanaro, taleinat, xtreak
Priority: normal Keywords: patch

Created on 2020-08-16 04:10 by bugsrep, last changed 2020-10-15 03:58 by bugsrep.

Files
File name Uploaded Description Edit
test_ssl_ubuntu.diff skip.montanaro, 2020-09-07 17:29
Pull Requests
URL Status Linked Edit
PR 22308 open christian.heimes, 2020-09-18 14:15
Messages (39)
msg375502 - (view) Author: Bug Reporter (bugsrep) Date: 2020-08-16 04:10
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vbk/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 217, in wrapper
    return func(*args, **kw)
  File "/home/vbk/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 3841, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1123)'
msg376501 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2020-09-07 15:23
Has any progress been made on the Ubuntu 20.04 test_ssl failures? Is there any consensus about it being a Python or Ubuntu problem?
msg376521 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2020-09-07 17:29
This skips the breaking tests (but doesn't actually fix anything).
msg376669 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-10 01:20
I don't know if it matters, but I started having this problem when I switched from Ubuntu 18.04 (native python3.7) to 20.04 (native python3.8.2). I specify --prefix to a folder in my home directory, but while running make test Ubuntu gives a system error which refers to Ubuntu's python. I don't know exactly at what test it happens, approximately in the middle, but it should not happen at all because the tests should only call the python compiled by me.
msg376688 - (view) Author: Vladyslav Bondar (Vladyslav.Bondar) Date: 2020-09-10 16:37
This will help to solve it

https://stackoverflow.com/questions/61568215/openssl-v1-1-1-ubuntu-20-tlsv1-no-protocols-available

But in my case I've defined:
MinProtocol = None
msg376690 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2020-09-10 18:22
@Vladyslav.Bondar I can't tell where you are suggesting MinProtocol should be set. I don't see that particular string in any .c, .h or .py file in the Python source.
msg376705 - (view) Author: Vladyslav Bondar (Vladyslav.Bondar) Date: 2020-09-11 09:10
This is about openssl configuration in Ubuntu. In the latest Ubuntu, they disabled TLS 1.0/1.1.

So to enable it back there is a workaround (taken from StackOverflow):

You should modify openssl config: /etc/ssl/openssl.cnf

You need to add this to the beginning of your config file:

openssl_conf = default_conf

And then this to the end:

[ default_conf ]

ssl_conf = ssl_sect

[ssl_sect]

system_default = ssl_default_sect

[ssl_default_sect]
MinProtocol = None
CipherString = DEFAULT:@SECLEVEL=1
msg376710 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2020-09-11 09:37
issue38815 also reported similar issue in test_min_max_version_mismatch.
msg376768 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-12 03:13
I followed Vladyslav Bondar's 2020-09-11 09:10:30 recommendations and it worked:
Tests result: SUCCESS

Thank you.

It is not clear though how Canonical built its python-3.8.2 which comes with Ubuntu-20.04. Does anyone know someone at Canonical to ask this question?
msg376817 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-13 02:34
Can test_ssl script determine which TLS versions are enabled in a particular Linux distribution and run tests only for enabled versions?
msg377048 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-17 09:29
I just ran into this too on Ubuntu 20.04.

I don't recommend changing /etc/ssl/openssl.cnf. Instead, make a local copy, for example at $HOME/cpython-dev-openssl.cnf, with the changes suggested by Vladyslav Bondar. Then run the tests with:

OPENSSL_CONF=$HOME/openssl-cpython-dev.cnf ./python -m test
msg377085 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-18 04:10
I confirm that Tal Einat's workaround also works.
msg377112 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2020-09-18 12:22
>
> I confirm that Tal Einat's workaround also works.
>

Should workarounds be required to successfully run the test suite? I
routinely unset PYTHONSTARTUP, but that's because I can and sometimes do
weird things to support interactive use. It concerns me a bit that any
system settings need to be overridden. That suggests to me that it will
eventually bite users at runtime.

>
msg377115 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-18 13:50
I think it is not a real solution, just a workaround. What do you think the solution should be?
msg377116 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-18 13:57
Could you please modify the test case to check for TLS 1.0 and run it Ubuntu?

    @requires_minimum_version
    @requires_tls_version('TLSv1_2')
    @requires_tls_version('TLSv1')
    def test_min_max_version_mismatch(self):

For Python 3.10 I'm planning to drop support for TLS 1.1 and earlier.
msg377117 - (view) Author: Skip Montanaro (skip.montanaro) * (Python triager) Date: 2020-09-18 14:08
> Could you please modify the test case to check for TLS 1.0 and run it Ubuntu?
>
>     @requires_minimum_version
>     @requires_tls_version('TLSv1_2')
>     @requires_tls_version('TLSv1')
>     def test_min_max_version_mismatch(self):

Confirmed. test_min_max_version_mismatch passes with those three decorators.
msg377145 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-19 01:36
On my system, it still fails.

Here is the content of modified Lib/test/test_ssl.py:

3827     @requires_minimum_version
3828     @requires_tls_version('TLSv1_2')
3829     @requires_tls_version('TLSv1')
3830     def test_min_max_version_mismatch(self):
3831         client_context, server_context, hostname = testing_context()
3832         # client 1.0, server 1.2 (mismatch)
3833         server_context.maximum_version = ssl.TLSVersion.TLSv1_2
3834         server_context.minimum_version = ssl.TLSVersion.TLSv1_2
3835         client_context.maximum_version = ssl.TLSVersion.TLSv1
3836         client_context.minimum_version = ssl.TLSVersion.TLSv1
3837         with ThreadedEchoServer(context=server_context) as server:
3838             with client_context.wrap_socket(socket.socket(),
3839                                             server_hostname=hostname) as s:
3840                 with self.assertRaises(ssl.SSLError) as e:
3841                     s.connect((HOST, server.port))
3842                 self.assertIn("alert", str(e.exception))
3843
3844     @requires_minimum_version
3845     @requires_tls_version('SSLv3')
msg377148 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-19 01:40
https://www.python.org/ftp/python/3.8.5/Python-3.8.5.tar.xz

...
======================================================================
ERROR: test_protocol_tlsv1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1 server with various client options
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 217, in wrapper
    return func(*args, **kw)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 3286, in test_protocol_tlsv1
    try_protocol_combo(ssl.PROTOCOL_TLSv1, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 2780, in try_protocol_combo
    stats = server_params_test(client_context, server_context,
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 2695, in server_params_test
    s.connect((HOST, server.port))
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL] internal error (_ssl.c:1123)

======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 217, in wrapper
    return func(*args, **kw)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 3302, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_TLSv1_1, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 2780, in try_protocol_combo
    stats = server_params_test(client_context, server_context,
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 2695, in server_params_test
    s.connect((HOST, server.port))
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL] internal error (_ssl.c:1123)

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 217, in wrapper
    return func(*args, **kw)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 217, in wrapper
    return func(*args, **kw)
  File "/home/bugrep/Downloads/Python-3.8.5/Lib/test/test_ssl.py", line 3842, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1123)'
msg377155 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-19 09:47
It's starting to look like a misconfiguration in either Ubuntu's OpenSSL build or your system. has_tls_version() checks compile time options and runtime configuration options. It should detect that TLS 1.1 and 1.0 are not available. "[SSL] internal error" also points to an unusual error condition that should never be triggered by these tests.

Please run this on your system:

import ssl
print((ssl.HAS_TLSv1, ssl.HAS_TLSv1_1, ssl.HAS_TLSv1_2, ssl.HAS_TLSv1_3))
print(ssl.SSLContext().maximum_version)
print(ssl.SSLContext().minimum_version)
msg377172 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-19 15:46
(True, True, True, True)
TLSVersion.MAXIMUM_SUPPORTED
TLSVersion.MINIMUM_SUPPORTED
msg377174 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-19 15:52
In case it's needed:
TLSVersion.MAXIMUM_SUPPORTED == -1
TLSVersion.MINIMUM_SUPPORTED == -2
msg377179 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-19 18:07
Likewise here on Ubuntu 20.04:

(True, True, True, True)
-1
-2
msg377221 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-20 17:02
Four times 'True' means that OpenSSL is compiled with TLS 1.0, 1,1, 1.2, and 1.3 support. SSLContext().minium_version == MINIMUM_SUPPORTED and maximum_version == MAXIMUM_SUPPORTED mean that no crypto policy setting or OpenSSL security level setting has modified the minimum and maximum version. TLS 1.0 and 1.1 connection should work.

But it's not working for you and some connection are even failing "internal error". This smells like a Debian/Ubuntu have patched OpenSSL with a buggy patch that breaks OpenSSL's internal API.
msg377224 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-20 17:47
Does "test_min_max_version_mismatch" pass on your system when you add

    client_context.set_ciphers("@SECLEVEL=1:HIGH")

?
msg377225 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 18:22
No, adding that after the first line of test_min_max_version_mismatch() still results in the same error:

ssl.SSLError: [SSL: UNSUPPORTED_PROTOCOL] unsupported protocol (_ssl.c:1122)
msg377228 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-20 18:39
What happens if you modify both contexts?
msg377232 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-20 19:19
Same.
msg377240 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-21 02:44
I added client_context.set_ciphers("@SECLEVEL=1:HIGH"), then added server_context.set_ciphers("@SECLEVEL=1:HIGH"). The test failed in both cases.

I did not have problem with python 3.7.x. in Ubuntu 18.04. I have just tried compiling 3.7.5 in Ubuntu 20.04 and test_ssl failed.

I also remember downloading openssl source code, compiling it, and using --with-openssl=DIR option with python 3.8.x in Ubuntu 20.04. I tried different versions of openssl (I did not edit any config files, just compiled) and test_ssl failed with all of them. Does it mean that Ubuntu's config files were still used even in this case?
msg377252 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-21 12:25
Did you set an rpath, reconfigure ldconfig, or LD_LIBRARY_PATH env var? If not then you compiled Python with an alternative OpenSSL installation but did not instruct the ld to load the alternative shared libraries. --with-openssl only modifies header and linker search, not dynamic loader options.

The command

    ldd $(find build -name '_ssl*.so')

will should you what shared OpenSSL libraries the dynamic linker will load.

I wrote a script to download and compile OpenSSL and then run Python's test suite with exactly that OpenSSL build:

    ./configure
    make
    ./python Tools/ssl/multissltests.py --openssl=1.1.1g
msg377297 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-22 03:57
1) I downloaded openssl-1.1.1g source code, compiled, exported LD_LIBRARY_PATH, did sudo ldconfig - ./python Lib/test/test_ssl.py now passes. If LD_LIBRARY_PATH is unset, it fails. Thank you for advice.


2) Tried the following independently from (1)
./configure
make
./python Tools/ssl/multissltests.py --openssl=1.1.1g

== Tests result: SUCCESS ==

All 15 tests OK.

Total duration: 1 min 21 sec
Tests result: SUCCESS
*** INFO 
Tests finished in 0:05:52.488541
Python:  3.8.5 (default, Sep 21 2020, 23:02:31) 
[GCC 9.3.0]
Executed all SSL tests.
OpenSSL / LibreSSL versions:
    * OpenSSL 1.1.1g
msg377403 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-23 16:05
Is it worth comparing openssl vanilla code and configs with Ubuntu's version?
msg377415 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-09-23 18:27
Yes, that would be useful. I suspect tls1.2-min-seclevel2.patch from the patch set https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu2 might be the cause of this issue.
msg377516 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-26 03:56
I downloaded Ubuntu's openssl_1.1.1f.orig.tar.gz and openssl_1.1.1f-1ubuntu2.debian.tar.xz from https://launchpad.net/ubuntu/+source/openssl/1.1.1f-1ubuntu2, but I did not know how to apply patches. In addition, too many files differ, so I could not understand what makes test_ssl fail. So I took a different approach.

In Ubuntu-20.04, "apt policy openssl" returned the version of the installed library: 1.1.1f-1ubuntu2. I ran "apt source openssl" to downloade the source code from Ubuntu. I compiled, tested and installed it.


If LD_LIBRARY_PATH is not set, ldd returns this:
  libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1
  libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

make test TESTOPTS="-v test_ssl":
  FAILED (errors=6, skipped=11)


If LD_LIBRARY_PATH is set to compiled openssl-1.1.1f-1ubuntu2:
  libssl.so.1.1 => /home/bugsrep/openssl-ubuntu2/lib/libssl.so.1.1 
  libcrypto.so.1.1 => /home/bugsrep/openssl-ubuntu2/lib/libcrypto.so.1.1

make test TESTOPTS="-v test_ssl":
  OK (skipped=11)
  == Tests result: SUCCESS ==
  1 test OK.
msg377517 - (view) Author: Bug Reporter (bugsrep) Date: 2020-09-26 04:18
Please note that test_ssl also passes if /etc/ssl/openssl.conf is modified per msg376705 by Vladyslav Bondar (with /usr/lib/x86_64-linux-gnu/libssl.so.1.1)
msg378089 - (view) Author: Bug Reporter (bugsrep) Date: 2020-10-06 04:38
Just tested python 3.9.0 - same issue.
msg378103 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-06 09:12
It sounds like a Debian/Ubuntu patch is breaking an assumption. Did somebody report the bug with Debian/Ubuntu maintainers of OpenSSL already?

Fedora also configures OpenSSL with minimum protocol version of TLS 1.2. The distribution does it in a slightly different way that makes the restriction discoverable and that is compatible with Python's test suite.
msg378440 - (view) Author: Bug Reporter (bugsrep) Date: 2020-10-11 17:21
I started by asking a question at https://askubuntu.com/questions/1281942/pythons-test-ssl-fails-starting-from-ubuntu-20-04-i-need-to-find-a-person-at-c
msg378510 - (view) Author: Bug Reporter (bugsrep) Date: 2020-10-12 17:31
I got an advice and posted the question at https://answers.launchpad.net/ubuntu/+source/openssl/+question/693423
msg378662 - (view) Author: Bug Reporter (bugsrep) Date: 2020-10-15 03:58
I reported a bug at https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878
History
Date User Action Args
2020-10-15 03:58:44bugsrepsetmessages: + msg378662
2020-10-12 17:31:19bugsrepsetmessages: + msg378510
2020-10-11 17:21:58bugsrepsetmessages: + msg378440
2020-10-06 09:12:09christian.heimessetmessages: + msg378103
2020-10-06 04:38:34bugsrepsetmessages: + msg378089
versions: + Python 3.9
2020-09-26 04:18:22bugsrepsetmessages: + msg377517
2020-09-26 03:56:13bugsrepsetmessages: + msg377516
2020-09-23 18:27:05christian.heimessetmessages: + msg377415
2020-09-23 16:05:17bugsrepsetmessages: + msg377403
2020-09-22 03:57:35bugsrepsetmessages: + msg377297
2020-09-21 12:25:39christian.heimessetmessages: + msg377252
2020-09-21 02:44:47bugsrepsetmessages: + msg377240
2020-09-20 19:19:35taleinatsetmessages: + msg377232
2020-09-20 18:39:36christian.heimessetmessages: + msg377228
2020-09-20 18:22:37taleinatsetmessages: + msg377225
2020-09-20 17:47:40christian.heimessetmessages: + msg377224
2020-09-20 17:02:11christian.heimessetmessages: + msg377221
2020-09-19 18:07:20taleinatsetmessages: + msg377179
2020-09-19 15:52:48bugsrepsetmessages: + msg377174
2020-09-19 15:46:05bugsrepsetmessages: + msg377172
2020-09-19 09:47:05christian.heimessetmessages: + msg377155
2020-09-19 01:40:55bugsrepsetmessages: + msg377148
2020-09-19 01:36:28bugsrepsetmessages: + msg377145
2020-09-18 14:15:35christian.heimessetstage: patch review
pull_requests: + pull_request21356
2020-09-18 14:08:37skip.montanarosetmessages: + msg377117
2020-09-18 13:57:56christian.heimessetmessages: + msg377116
2020-09-18 13:50:35bugsrepsetmessages: + msg377115
2020-09-18 12:22:11skip.montanarosetmessages: + msg377112
2020-09-18 04:10:00bugsrepsetmessages: + msg377085
versions: - Python 3.9, Python 3.10
2020-09-17 09:29:29taleinatsetversions: + Python 3.9, Python 3.10
2020-09-17 09:29:20taleinatsetnosy: + taleinat
messages: + msg377048
2020-09-13 02:34:55bugsrepsetmessages: + msg376817
2020-09-12 03:13:21bugsrepsetmessages: + msg376768
versions: + Python 3.8, - Python 3.10
2020-09-11 09:37:46xtreaksetnosy: + xtreak
messages: + msg376710
2020-09-11 09:10:30Vladyslav.Bondarsetmessages: + msg376705
2020-09-10 18:22:05skip.montanarosetmessages: + msg376690
2020-09-10 16:37:43Vladyslav.Bondarsetnosy: + Vladyslav.Bondar
messages: + msg376688
2020-09-10 01:20:05bugsrepsetmessages: + msg376669
2020-09-07 17:29:44skip.montanarosetfiles: + test_ssl_ubuntu.diff
keywords: + patch
messages: + msg376521

versions: + Python 3.10, - Python 3.8
2020-09-07 15:23:41skip.montanarosetnosy: + skip.montanaro
messages: + msg376501
2020-08-16 04:10:49bugsrepcreate