classification
Title: test_ssl: test_min_max_version() fails on FreeBSD and Fedora
Type: Stage: commit review
Components: SSL, Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, koobs, miss-islington, ned.deily, pablogsal, sumagnadas, taleinat, vstinner
Priority: normal Keywords: patch

Created on 2019-11-15 16:54 by vstinner, last changed 2020-09-26 20:00 by sumagnadas.

Pull Requests
URL Status Linked Edit
PR 17437 merged torsava, 2019-12-02 15:54
PR 17438 merged miss-islington, 2019-12-02 16:16
PR 17439 merged miss-islington, 2019-12-02 16:16
Messages (25)
msg356691 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-15 16:54
Fail with OpenSSL 1.1.1d  10 Sep 2019 on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/371/builds/78

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1238, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3840, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'


SSL infos from pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1d  10 Sep 2019
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 4, 15)
ssl.OP_ALL: 0x80000054
ssl.OP_NO_TLSv1_1: 0x10000000

ssl.SSLContext.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.SSLContext.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.SSLContext.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.SSLContext.protocol: _SSLMethod.PROTOCOL_TLS
ssl.SSLContext.verify_mode: VerifyMode.CERT_NONE

ssl.default_https_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.default_https_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.default_https_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.default_https_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.default_https_context.verify_mode: VerifyMode.CERT_REQUIRED

ssl.stdlib_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.stdlib_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.stdlib_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.stdlib_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.stdlib_context.verify_mode: VerifyMode.CERT_NONE
msg356692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-15 16:55
Same failure on AMD64 FreeBSD Shared 3.8:
https://buildbot.python.org/all/#/builders/374/builds/32

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
msg356857 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-18 11:01
And the last one, AMD64 FreeBSD Shared 3.7:
https://buildbot.python.org/all/#/builders/367/builds/25

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
msg357705 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:07
Similar issue in Fedora: https://src.fedoraproject.org/rpms/python3/pull-request/155
msg357706 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:10
Debian and Fedora are already using OPENSSL_CONF=/non-existing-file workaround.

IMHO test_ssl should be fixed instead. test_ssl should not make assumptions on min/max TLS version.
msg357707 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:28
Previous attempt:
https://github.com/python/cpython/pull/11508#issuecomment-453186969

Old Fedora issue, worked around in the Fedora buildbot configuration: bpo-35045.
msg357710 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:42
I marked bpo-38954 as duplicate of this issue. Copy of its message:
---
 This issue is probably duplicate of some other, but I decided to make one new one due to the fact that this affect all Fedora build bots.


test test_ssl failed
test_timeout_connect_ex (test.test_ssl.NetworkedTests) ... ok
======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-fedora-rawhide-x86_64.lto-pgo/build/Lib/test/test_ssl.py", line 1207, in test_min_max_version
    self.assertEqual(
AssertionError: <TLSVersion.TLSv1_3: 772> != <TLSVersion.MAXIMUM_SUPPORTED: -1>
----------------------------------------------------------------------
Ran 161 tests in 2.681s
FAILED (failures=1, skipped=11)
1 test failed again:
    test_ssl

Example failure:

https://buildbot.python.org/all/#/builders/222
---
msg357711 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 13:55
Raising the priority to 'critical' as this is masking all other problems on a significant number of buildbots.

Christian, could you take a look?
msg357712 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-12-02 15:32
Tomas Orsava is going to provide a PR within the next hour.
msg357715 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:15
New changeset 34864d1cffdbfc620f8517dab9a68ae9a37b8c53 by Miss Islington (bot) (torsava) in branch 'master':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/34864d1cffdbfc620f8517dab9a68ae9a37b8c53
msg357716 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 16:18
Testing for the latest PR happening here:

https://buildbot.python.org/all/#/builders?tags=%2Bstable&tags=%2Bcustom
msg357717 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:34
New changeset 9e728806d03fff8fa9e75159e567b2b4e040971b by Miss Islington (bot) in branch '3.8':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/9e728806d03fff8fa9e75159e567b2b4e040971b
msg357718 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:35
New changeset 1f4f28cbf3ca4c7f85b41b1f7b9455a375415cc0 by Miss Islington (bot) in branch '3.7':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/1f4f28cbf3ca4c7f85b41b1f7b9455a375415cc0
msg357719 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 17:35
All Fedora tests are successful, so I will de-escalate the priority of the issue. Thank you very much Christian and Tomas Orsava!
msg357792 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-04 10:57
test_ssl still fails on FreeBSD:

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1244, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3846, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'
msg357794 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-12-04 11:34
In case it's relevant, I note the following:

The ssl module is built with -I/usr/local/include in the compile line.

Leaving aside:

 - the warts of the build system, and 
 - the lack of specific --with-foo=<path> semantics to allow providing well(narrowly)-scoped include/library paths
 - how and whether the default python build using -I/usr/local/include is 'good' or not ...

it means that the build is finding openssl111 on FreeBSD provided by ports, not base (base also includes openssl 1.1.1 @ /usr/include|lib)

More specifically, our openssl111 port happens to have many options that allow enabling/disabling various software features, including encryption algorithms and protocol versions

The current (installed) build of openssl111 on the buildbot worker that is failing, has the following options set/unset:

OPTIONS_FILE_UNSET+=CT
OPTIONS_FILE_UNSET+=MAN3
OPTIONS_FILE_UNSET+=RFC3779
OPTIONS_FILE_SET+=SHARED
OPTIONS_FILE_UNSET+=ZLIB
OPTIONS_FILE_UNSET+=ARIA
OPTIONS_FILE_UNSET+=DES
OPTIONS_FILE_UNSET+=GOST
OPTIONS_FILE_UNSET+=IDEA
OPTIONS_FILE_UNSET+=SM2
OPTIONS_FILE_UNSET+=SM3
OPTIONS_FILE_UNSET+=SM4
OPTIONS_FILE_UNSET+=RC2
OPTIONS_FILE_UNSET+=RC4
OPTIONS_FILE_UNSET+=RC5
OPTIONS_FILE_UNSET+=MD2
OPTIONS_FILE_UNSET+=MD4
OPTIONS_FILE_UNSET+=MDC2
OPTIONS_FILE_UNSET+=RMD160
OPTIONS_FILE_SET+=ASM
OPTIONS_FILE_SET+=SSE2
OPTIONS_FILE_SET+=THREADS
OPTIONS_FILE_SET+=EC
OPTIONS_FILE_SET+=NEXTPROTONEG
OPTIONS_FILE_UNSET+=SCTP
OPTIONS_FILE_UNSET+=SSL3
OPTIONS_FILE_UNSET+=TLS1
OPTIONS_FILE_UNSET+=TLS1_1
OPTIONS_FILE_SET+=TLS1_2

In summary, this build only has TLS 1.2 and TLS 1.3 enabled (but with other various things disabled, not the defaults of the upstream openssl build), and may be indicated in this test failure, possibly only one example of many other similar issues of the same class, something like "tests assume certain features"

Relatedly, this method of disabling various default options in openssl, is how a number [1] of issues in the Python cryptography package were found:

https://github.com/pyca/cryptography/issues/5065

* Another issue is a build failure with NPN disabled, yet to be reported
msg357959 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-07 00:41
@koobs, FWIW, unlike for most other third-party libs, there now is a configure option to allow specifying the path to the desired OpenSSL version, actually two ways.  Either use the --with-openssl= option to ./configure. Otherwise, if pkg-config info is availble, ./configure will try to use that for OpenSSL.
msg358314 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-12-13 01:12
I had to rebuild the openssl (1.1.x) port on the worker that had test_ssl failing in order to bring it back to green, so as not to hide new test failures.

If/when someone is able to produce a fix for the failing test, I am happy to revert the openssl build to its prior (failing config) state at any time to verify the test fix.
msg358693 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-12-20 02:07
@Christian As current assignee, are you able to produce a test that fixes the remaining issue (per msg357792)? I can rebuild OpenSSL on the worker at your direction at any time to make the test fail again or provide you with an SSH account to assist
msg369252 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 15:41
I'm not sure why, but test_ssl does not pass on FreeBSD and Fedora, so I close the issue.
msg369254 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-18 15:43
Please don't close tickets assigned to an owner without consent.
msg369260 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 16:13
> Please don't close tickets assigned to an owner without consent.

Can you please elaborate why you reopened the issue?

The initial "test_ssl: test_min_max_version() fails on FreeBSD and Fedora" issue is now fixed.

For example, test_ssl pass on AMD64 Fedora Stable LTO 3.x buildbot and it no longer uses the OPENSSL_CONF=/non-existing-file workaround.
https://buildbot.python.org/all/#/builders/112/builds/850
"ssl.OPENSSL_VERSION: OpenSSL 1.1.1g FIPS  21 Apr 2020"

Same for AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/152/builds/852
"ssl.OPENSSL_VERSION: OpenSSL 1.1.1g  21 Apr 2020"

If there are remaining new issues, I would prefer to open new issues.
msg369272 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-05-18 16:37
If you think that a ticket no longer apply, please use the pending status and give me a chance to verify the result. I haven't looked into the issue and I don't have time to do it right now.
msg377049 - (view) Author: Tal Einat (taleinat) * (Python committer) Date: 2020-09-17 09:33
Ping?
msg377541 - (view) Author: Sumagna Das (sumagnadas) Date: 2020-09-26 19:58
still getting this issue on Ubuntu 20.04 with the latest master checkout of cpython
History
Date User Action Args
2020-09-26 20:00:21sumagnadassetversions: + Python 3.9, - Python 3.10
2020-09-26 19:58:58sumagnadassetnosy: + sumagnadas

messages: + msg377541
versions: + Python 3.10, - Python 3.9
2020-09-17 09:33:23taleinatsetstatus: pending -> open
nosy: + taleinat
messages: + msg377049

2020-05-18 16:37:32christian.heimessetstatus: open -> pending

messages: + msg369272
stage: resolved -> commit review
2020-05-18 16:13:30vstinnersetmessages: + msg369260
2020-05-18 15:43:23christian.heimessetstatus: closed -> open
resolution: fixed ->
messages: + msg369254
2020-05-18 15:41:45vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg369252

stage: needs patch -> resolved
2019-12-20 02:07:46koobssetmessages: + msg358693
stage: patch review -> needs patch
2019-12-13 01:12:55koobssetmessages: + msg358314
2019-12-07 00:41:37ned.deilysetnosy: + ned.deily
messages: + msg357959
2019-12-04 11:34:40koobssetnosy: + koobs
messages: + msg357794
2019-12-04 10:57:29vstinnersetmessages: + msg357792
2019-12-02 17:35:05pablogsalsetpriority: critical -> normal

messages: + msg357719
2019-12-02 16:35:54miss-islingtonsetmessages: + msg357718
2019-12-02 16:34:48miss-islingtonsetmessages: + msg357717
2019-12-02 16:18:09pablogsalsetmessages: + msg357716
2019-12-02 16:16:17miss-islingtonsetpull_requests: + pull_request16920
2019-12-02 16:16:10miss-islingtonsetpull_requests: + pull_request16919
2019-12-02 16:15:46miss-islingtonsetnosy: + miss-islington
messages: + msg357715
2019-12-02 15:54:09torsavasetkeywords: + patch
stage: patch review
pull_requests: + pull_request16917
2019-12-02 15:32:21christian.heimessetmessages: + msg357712
2019-12-02 13:55:03pablogsalsetpriority: normal -> critical
nosy: + pablogsal
messages: + msg357711

2019-12-02 13:42:23vstinnersetmessages: + msg357710
title: test_ssl: test_min_max_version() fails on AMD64 FreeBSD Shared 3.x -> test_ssl: test_min_max_version() fails on FreeBSD and Fedora
2019-12-02 13:41:41vstinnerlinkissue38954 superseder
2019-12-02 13:28:42vstinnersetmessages: + msg357707
2019-12-02 13:10:55vstinnersetmessages: + msg357706
2019-12-02 13:07:32vstinnersetmessages: + msg357705
2019-11-18 11:01:21vstinnersetmessages: + msg356857
2019-11-15 16:55:11vstinnersetmessages: + msg356692
2019-11-15 16:54:20vstinnercreate