test_ssl: test_min_max_version() fails on FreeBSD and Fedora #82996

vstinner · 2019-11-15T16:54:21Z

BPO	38815
Nosy	@vstinner, @taleinat, @tiran, @ned-deily, @koobs, @pablogsal, @miss-islington, @sumagnadas
PRs	bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) #17437 [3.8] bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437) #17438 [3.7] bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437) #17439

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2021-04-17.18:19:39.273>
created_at = <Date 2019-11-15.16:54:20.894>
labels = ['expert-SSL', 'tests', '3.9']
title = 'test_ssl: test_min_max_version() fails on FreeBSD and Fedora'
updated_at = <Date 2021-04-17.18:19:39.272>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-04-17.18:19:39.272>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2021-04-17.18:19:39.273>
closer = 'christian.heimes'
components = ['Tests', 'SSL']
creation = <Date 2019-11-15.16:54:20.894>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 38815
keywords = ['patch']
message_count = 26.0
messages = ['356691', '356692', '356857', '357705', '357706', '357707', '357710', '357711', '357712', '357715', '357716', '357717', '357718', '357719', '357792', '357794', '357959', '358314', '358693', '369252', '369254', '369260', '369272', '377049', '377541', '391298']
nosy_count = 8.0
nosy_names = ['vstinner', 'taleinat', 'christian.heimes', 'ned.deily', 'koobs', 'pablogsal', 'miss-islington', 'sumagnadas']
pr_nums = ['17437', '17438', '17439']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue38815'
versions = ['Python 3.9']

vstinner · 2019-11-15T16:54:21Z

Fail with OpenSSL 1.1.1d 10 Sep 2019 on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/371/builds/78

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1238, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3840, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'

SSL infos from pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1d 10 Sep 2019
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 4, 15)
ssl.OP_ALL: 0x80000054
ssl.OP_NO_TLSv1_1: 0x10000000

ssl.SSLContext.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.SSLContext.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.SSLContext.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.SSLContext.protocol: _SSLMethod.PROTOCOL_TLS
ssl.SSLContext.verify_mode: VerifyMode.CERT_NONE

ssl.default_https_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.default_https_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.default_https_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.default_https_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.default_https_context.verify_mode: VerifyMode.CERT_REQUIRED

ssl.stdlib_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.stdlib_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.stdlib_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.stdlib_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.stdlib_context.verify_mode: VerifyMode.CERT_NONE

vstinner · 2019-11-15T16:55:12Z

Same failure on AMD64 FreeBSD Shared 3.8:
https://buildbot.python.org/all/#/builders/374/builds/32

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)

vstinner · 2019-11-18T11:01:22Z

And the last one, AMD64 FreeBSD Shared 3.7:
https://buildbot.python.org/all/#/builders/367/builds/25

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)

vstinner · 2019-12-02T13:07:32Z

Similar issue in Fedora: https://src.fedoraproject.org/rpms/python3/pull-request/155

vstinner · 2019-12-02T13:10:56Z

Debian and Fedora are already using OPENSSL_CONF=/non-existing-file workaround.

IMHO test_ssl should be fixed instead. test_ssl should not make assumptions on min/max TLS version.

vstinner · 2019-12-02T13:28:43Z

Previous attempt:
#11508 (comment)

Old Fedora issue, worked around in the Fedora buildbot configuration: bpo-35045.

vstinner · 2019-12-02T13:42:24Z

I marked bpo-38954 as duplicate of this issue. Copy of its message:
---
This issue is probably duplicate of some other, but I decided to make one new one due to the fact that this affect all Fedora build bots.

test test_ssl failed
test_timeout_connect_ex (test.test_ssl.NetworkedTests) ... ok
======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-fedora-rawhide-x86_64.lto-pgo/build/Lib/test/test_ssl.py", line 1207, in test_min_max_version
    self.assertEqual(
AssertionError: <TLSVersion.TLSv1_3: 772> != <TLSVersion.MAXIMUM_SUPPORTED: -1>

Ran 161 tests in 2.681s
FAILED (failures=1, skipped=11)
1 test failed again:
test_ssl

Example failure:

https://buildbot.python.org/all/#/builders/222
---

pablogsal · 2019-12-02T13:55:03Z

Raising the priority to 'critical' as this is masking all other problems on a significant number of buildbots.

Christian, could you take a look?

tiran · 2019-12-02T15:32:21Z

Tomas Orsava is going to provide a PR within the next hour.

miss-islington · 2019-12-02T16:15:46Z

New changeset 34864d1 by Miss Islington (bot) (torsava) in branch 'master':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
34864d1

pablogsal · 2019-12-02T16:18:09Z

Testing for the latest PR happening here:

https://buildbot.python.org/all/#/builders?tags=%2Bstable&tags=%2Bcustom

miss-islington · 2019-12-02T16:34:48Z

New changeset 9e72880 by Miss Islington (bot) in branch '3.8':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
9e72880

miss-islington · 2019-12-02T16:35:54Z

New changeset 1f4f28c by Miss Islington (bot) in branch '3.7':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
1f4f28c

pablogsal · 2019-12-02T17:35:06Z

All Fedora tests are successful, so I will de-escalate the priority of the issue. Thank you very much Christian and Tomas Orsava!

vstinner · 2019-12-04T10:57:29Z

test_ssl still fails on FreeBSD:

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1244, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3846, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'

koobs · 2019-12-04T11:34:40Z

In case it's relevant, I note the following:

The ssl module is built with -I/usr/local/include in the compile line.

Leaving aside:

the warts of the build system, and
the lack of specific --with-foo=<path> semantics to allow providing well(narrowly)-scoped include/library paths
how and whether the default python build using -I/usr/local/include is 'good' or not ...

it means that the build is finding openssl111 on FreeBSD provided by ports, not base (base also includes openssl 1.1.1 @ /usr/include|lib)

More specifically, our openssl111 port happens to have many options that allow enabling/disabling various software features, including encryption algorithms and protocol versions

The current (installed) build of openssl111 on the buildbot worker that is failing, has the following options set/unset:

OPTIONS_FILE_UNSET+=CT
OPTIONS_FILE_UNSET+=MAN3
OPTIONS_FILE_UNSET+=RFC3779
OPTIONS_FILE_SET+=SHARED
OPTIONS_FILE_UNSET+=ZLIB
OPTIONS_FILE_UNSET+=ARIA
OPTIONS_FILE_UNSET+=DES
OPTIONS_FILE_UNSET+=GOST
OPTIONS_FILE_UNSET+=IDEA
OPTIONS_FILE_UNSET+=SM2
OPTIONS_FILE_UNSET+=SM3
OPTIONS_FILE_UNSET+=SM4
OPTIONS_FILE_UNSET+=RC2
OPTIONS_FILE_UNSET+=RC4
OPTIONS_FILE_UNSET+=RC5
OPTIONS_FILE_UNSET+=MD2
OPTIONS_FILE_UNSET+=MD4
OPTIONS_FILE_UNSET+=MDC2
OPTIONS_FILE_UNSET+=RMD160
OPTIONS_FILE_SET+=ASM
OPTIONS_FILE_SET+=SSE2
OPTIONS_FILE_SET+=THREADS
OPTIONS_FILE_SET+=EC
OPTIONS_FILE_SET+=NEXTPROTONEG
OPTIONS_FILE_UNSET+=SCTP
OPTIONS_FILE_UNSET+=SSL3
OPTIONS_FILE_UNSET+=TLS1
OPTIONS_FILE_UNSET+=TLS1_1
OPTIONS_FILE_SET+=TLS1_2

In summary, this build only has TLS 1.2 and TLS 1.3 enabled (but with other various things disabled, not the defaults of the upstream openssl build), and may be indicated in this test failure, possibly only one example of many other similar issues of the same class, something like "tests assume certain features"

Relatedly, this method of disabling various default options in openssl, is how a number [1] of issues in the Python cryptography package were found:

pyca/cryptography#5065

Another issue is a build failure with NPN disabled, yet to be reported

ned-deily · 2019-12-07T00:41:37Z

@koobs, FWIW, unlike for most other third-party libs, there now is a configure option to allow specifying the path to the desired OpenSSL version, actually two ways. Either use the --with-openssl= option to ./configure. Otherwise, if pkg-config info is availble, ./configure will try to use that for OpenSSL.

koobs · 2019-12-13T01:12:56Z

I had to rebuild the openssl (1.1.x) port on the worker that had test_ssl failing in order to bring it back to green, so as not to hide new test failures.

If/when someone is able to produce a fix for the failing test, I am happy to revert the openssl build to its prior (failing config) state at any time to verify the test fix.

koobs · 2019-12-20T02:07:47Z

@christian As current assignee, are you able to produce a test that fixes the remaining issue (per msg357792)? I can rebuild OpenSSL on the worker at your direction at any time to make the test fail again or provide you with an SSH account to assist

vstinner · 2020-05-18T15:41:45Z

I'm not sure why, but test_ssl does not pass on FreeBSD and Fedora, so I close the issue.

tiran · 2020-05-18T15:43:24Z

Please don't close tickets assigned to an owner without consent.

vstinner · 2020-05-18T16:13:30Z

Please don't close tickets assigned to an owner without consent.

Can you please elaborate why you reopened the issue?

The initial "test_ssl: test_min_max_version() fails on FreeBSD and Fedora" issue is now fixed.

For example, test_ssl pass on AMD64 Fedora Stable LTO 3.x buildbot and it no longer uses the OPENSSL_CONF=/non-existing-file workaround.
https://buildbot.python.org/all/#/builders/112/builds/850
"ssl.OPENSSL_VERSION: OpenSSL 1.1.1g FIPS 21 Apr 2020"

Same for AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/152/builds/852
"ssl.OPENSSL_VERSION: OpenSSL 1.1.1g 21 Apr 2020"

If there are remaining new issues, I would prefer to open new issues.

tiran · 2020-05-18T16:37:32Z

If you think that a ticket no longer apply, please use the pending status and give me a chance to verify the result. I haven't looked into the issue and I don't have time to do it right now.

taleinat · 2020-09-17T09:33:23Z

Ping?

sumagnadas · 2020-09-26T19:58:58Z

still getting this issue on Ubuntu 20.04 with the latest master checkout of cpython

tiran · 2021-04-17T18:19:39Z

I haven't seen the problem in a while.

vstinner added the 3.9 only security fixes label Nov 15, 2019

vstinner assigned tiran Nov 15, 2019

vstinner added tests Tests in the Lib/test dir topic-SSL labels Nov 15, 2019

vstinner changed the title ~~test_ssl: test_min_max_version() fails on AMD64 FreeBSD Shared 3.x~~ test_ssl: test_min_max_version() fails on FreeBSD and Fedora Dec 2, 2019

vstinner closed this as completed May 18, 2020

tiran reopened this May 18, 2020

sumagnadas mannequin added 3.10 only security fixes 3.9 only security fixes and removed 3.9 only security fixes 3.10 only security fixes labels Sep 26, 2020

tiran closed this as completed Apr 17, 2021

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_ssl: test_min_max_version() fails on FreeBSD and Fedora #82996

test_ssl: test_min_max_version() fails on FreeBSD and Fedora #82996

vstinner commented Nov 15, 2019

vstinner commented Nov 15, 2019

vstinner commented Nov 15, 2019

vstinner commented Nov 18, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

pablogsal commented Dec 2, 2019

tiran commented Dec 2, 2019

miss-islington commented Dec 2, 2019

pablogsal commented Dec 2, 2019

miss-islington commented Dec 2, 2019

miss-islington commented Dec 2, 2019

pablogsal commented Dec 2, 2019

vstinner commented Dec 4, 2019

koobs commented Dec 4, 2019

ned-deily commented Dec 7, 2019

koobs commented Dec 13, 2019

koobs commented Dec 20, 2019

vstinner commented May 18, 2020

tiran commented May 18, 2020

vstinner commented May 18, 2020

tiran commented May 18, 2020

taleinat commented Sep 17, 2020

sumagnadas mannequin commented Sep 26, 2020

tiran commented Apr 17, 2021

test_ssl: test_min_max_version() fails on FreeBSD and Fedora #82996

test_ssl: test_min_max_version() fails on FreeBSD and Fedora #82996

Comments

vstinner commented Nov 15, 2019

vstinner commented Nov 15, 2019

vstinner commented Nov 15, 2019

vstinner commented Nov 18, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

vstinner commented Dec 2, 2019

pablogsal commented Dec 2, 2019

tiran commented Dec 2, 2019

miss-islington commented Dec 2, 2019

pablogsal commented Dec 2, 2019

miss-islington commented Dec 2, 2019

miss-islington commented Dec 2, 2019

pablogsal commented Dec 2, 2019

vstinner commented Dec 4, 2019

koobs commented Dec 4, 2019

ned-deily commented Dec 7, 2019

koobs commented Dec 13, 2019

koobs commented Dec 20, 2019

vstinner commented May 18, 2020

tiran commented May 18, 2020

vstinner commented May 18, 2020

tiran commented May 18, 2020

taleinat commented Sep 17, 2020

sumagnadas mannequin commented Sep 26, 2020

tiran commented Apr 17, 2021