classification
Title: test_ssl: test_min_max_version() fails on FreeBSD and Fedora
Type: Stage: patch review
Components: SSL, Tests Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, koobs, miss-islington, ned.deily, pablogsal, vstinner
Priority: normal Keywords: patch

Created on 2019-11-15 16:54 by vstinner, last changed 2019-12-07 00:41 by ned.deily.

Pull Requests
URL Status Linked Edit
PR 17437 merged torsava, 2019-12-02 15:54
PR 17438 merged miss-islington, 2019-12-02 16:16
PR 17439 merged miss-islington, 2019-12-02 16:16
Messages (17)
msg356691 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-15 16:54
Fail with OpenSSL 1.1.1d  10 Sep 2019 on AMD64 FreeBSD Shared 3.x:
https://buildbot.python.org/all/#/builders/371/builds/78

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1238, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3840, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'


SSL infos from pythoninfo:

ssl.HAS_SNI: True
ssl.OPENSSL_VERSION: OpenSSL 1.1.1d  10 Sep 2019
ssl.OPENSSL_VERSION_INFO: (1, 1, 1, 4, 15)
ssl.OP_ALL: 0x80000054
ssl.OP_NO_TLSv1_1: 0x10000000

ssl.SSLContext.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.SSLContext.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.SSLContext.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.SSLContext.protocol: _SSLMethod.PROTOCOL_TLS
ssl.SSLContext.verify_mode: VerifyMode.CERT_NONE

ssl.default_https_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.default_https_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.default_https_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.default_https_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.default_https_context.verify_mode: VerifyMode.CERT_REQUIRED

ssl.stdlib_context.maximum_version: TLSVersion.MAXIMUM_SUPPORTED
ssl.stdlib_context.minimum_version: TLSVersion.MINIMUM_SUPPORTED
ssl.stdlib_context.options: Options.OP_ALL|OP_NO_SSLv3|OP_CIPHER_SERVER_PREFERENCE|OP_ENABLE_MIDDLEBOX_COMPAT|OP_NO_COMPRESSION
ssl.stdlib_context.protocol: _SSLMethod.PROTOCOL_TLS
ssl.stdlib_context.verify_mode: VerifyMode.CERT_NONE
msg356692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-15 16:55
Same failure on AMD64 FreeBSD Shared 3.8:
https://buildbot.python.org/all/#/builders/374/builds/32

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
msg356857 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-11-18 11:01
And the last one, AMD64 FreeBSD Shared 3.7:
https://buildbot.python.org/all/#/builders/367/builds/25

FAIL: test_min_max_version (test.test_ssl.ContextTests)
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
msg357705 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:07
Similar issue in Fedora: https://src.fedoraproject.org/rpms/python3/pull-request/155
msg357706 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:10
Debian and Fedora are already using OPENSSL_CONF=/non-existing-file workaround.

IMHO test_ssl should be fixed instead. test_ssl should not make assumptions on min/max TLS version.
msg357707 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:28
Previous attempt:
https://github.com/python/cpython/pull/11508#issuecomment-453186969

Old Fedora issue, worked around in the Fedora buildbot configuration: bpo-35045.
msg357710 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-02 13:42
I marked bpo-38954 as duplicate of this issue. Copy of its message:
---
 This issue is probably duplicate of some other, but I decided to make one new one due to the fact that this affect all Fedora build bots.


test test_ssl failed
test_timeout_connect_ex (test.test_ssl.NetworkedTests) ... ok
======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.8.cstratak-fedora-rawhide-x86_64.lto-pgo/build/Lib/test/test_ssl.py", line 1207, in test_min_max_version
    self.assertEqual(
AssertionError: <TLSVersion.TLSv1_3: 772> != <TLSVersion.MAXIMUM_SUPPORTED: -1>
----------------------------------------------------------------------
Ran 161 tests in 2.681s
FAILED (failures=1, skipped=11)
1 test failed again:
    test_ssl

Example failure:

https://buildbot.python.org/all/#/builders/222
---
msg357711 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 13:55
Raising the priority to 'critical' as this is masking all other problems on a significant number of buildbots.

Christian, could you take a look?
msg357712 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2019-12-02 15:32
Tomas Orsava is going to provide a PR within the next hour.
msg357715 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:15
New changeset 34864d1cffdbfc620f8517dab9a68ae9a37b8c53 by Miss Islington (bot) (torsava) in branch 'master':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/34864d1cffdbfc620f8517dab9a68ae9a37b8c53
msg357716 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 16:18
Testing for the latest PR happening here:

https://buildbot.python.org/all/#/builders?tags=%2Bstable&tags=%2Bcustom
msg357717 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:34
New changeset 9e728806d03fff8fa9e75159e567b2b4e040971b by Miss Islington (bot) in branch '3.8':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/9e728806d03fff8fa9e75159e567b2b4e040971b
msg357718 - (view) Author: miss-islington (miss-islington) Date: 2019-12-02 16:35
New changeset 1f4f28cbf3ca4c7f85b41b1f7b9455a375415cc0 by Miss Islington (bot) in branch '3.7':
bpo-38815: Accept TLSv3 default in min max test (GH-NNNN) (GH-17437)
https://github.com/python/cpython/commit/1f4f28cbf3ca4c7f85b41b1f7b9455a375415cc0
msg357719 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-12-02 17:35
All Fedora tests are successful, so I will de-escalate the priority of the issue. Thank you very much Christian and Tomas Orsava!
msg357792 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-12-04 10:57
test_ssl still fails on FreeBSD:

======================================================================
FAIL: test_min_max_version (test.test_ssl.ContextTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 1244, in test_min_max_version
    self.assertIn(
AssertionError: <TLSVersion.TLSv1_2: 771> not found in {<TLSVersion.SSLv3: 768>, <TLSVersion.TLSv1: 769>}

======================================================================
FAIL: test_min_max_version_mismatch (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 220, in wrapper
    return func(*args, **kw)
  File "/usr/home/buildbot/python/3.x.koobs-freebsd-564d/build/Lib/test/test_ssl.py", line 3846, in test_min_max_version_mismatch
    self.assertIn("alert", str(e.exception))
AssertionError: 'alert' not found in '[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1108)'
msg357794 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2019-12-04 11:34
In case it's relevant, I note the following:

The ssl module is built with -I/usr/local/include in the compile line.

Leaving aside:

 - the warts of the build system, and 
 - the lack of specific --with-foo=<path> semantics to allow providing well(narrowly)-scoped include/library paths
 - how and whether the default python build using -I/usr/local/include is 'good' or not ...

it means that the build is finding openssl111 on FreeBSD provided by ports, not base (base also includes openssl 1.1.1 @ /usr/include|lib)

More specifically, our openssl111 port happens to have many options that allow enabling/disabling various software features, including encryption algorithms and protocol versions

The current (installed) build of openssl111 on the buildbot worker that is failing, has the following options set/unset:

OPTIONS_FILE_UNSET+=CT
OPTIONS_FILE_UNSET+=MAN3
OPTIONS_FILE_UNSET+=RFC3779
OPTIONS_FILE_SET+=SHARED
OPTIONS_FILE_UNSET+=ZLIB
OPTIONS_FILE_UNSET+=ARIA
OPTIONS_FILE_UNSET+=DES
OPTIONS_FILE_UNSET+=GOST
OPTIONS_FILE_UNSET+=IDEA
OPTIONS_FILE_UNSET+=SM2
OPTIONS_FILE_UNSET+=SM3
OPTIONS_FILE_UNSET+=SM4
OPTIONS_FILE_UNSET+=RC2
OPTIONS_FILE_UNSET+=RC4
OPTIONS_FILE_UNSET+=RC5
OPTIONS_FILE_UNSET+=MD2
OPTIONS_FILE_UNSET+=MD4
OPTIONS_FILE_UNSET+=MDC2
OPTIONS_FILE_UNSET+=RMD160
OPTIONS_FILE_SET+=ASM
OPTIONS_FILE_SET+=SSE2
OPTIONS_FILE_SET+=THREADS
OPTIONS_FILE_SET+=EC
OPTIONS_FILE_SET+=NEXTPROTONEG
OPTIONS_FILE_UNSET+=SCTP
OPTIONS_FILE_UNSET+=SSL3
OPTIONS_FILE_UNSET+=TLS1
OPTIONS_FILE_UNSET+=TLS1_1
OPTIONS_FILE_SET+=TLS1_2

In summary, this build only has TLS 1.2 and TLS 1.3 enabled (but with other various things disabled, not the defaults of the upstream openssl build), and may be indicated in this test failure, possibly only one example of many other similar issues of the same class, something like "tests assume certain features"

Relatedly, this method of disabling various default options in openssl, is how a number [1] of issues in the Python cryptography package were found:

https://github.com/pyca/cryptography/issues/5065

* Another issue is a build failure with NPN disabled, yet to be reported
msg357959 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-12-07 00:41
@koobs, FWIW, unlike for most other third-party libs, there now is a configure option to allow specifying the path to the desired OpenSSL version, actually two ways.  Either use the --with-openssl= option to ./configure. Otherwise, if pkg-config info is availble, ./configure will try to use that for OpenSSL.
History
Date User Action Args
2019-12-07 00:41:37ned.deilysetnosy: + ned.deily
messages: + msg357959
2019-12-04 11:34:40koobssetnosy: + koobs
messages: + msg357794
2019-12-04 10:57:29vstinnersetmessages: + msg357792
2019-12-02 17:35:05pablogsalsetpriority: critical -> normal

messages: + msg357719
2019-12-02 16:35:54miss-islingtonsetmessages: + msg357718
2019-12-02 16:34:48miss-islingtonsetmessages: + msg357717
2019-12-02 16:18:09pablogsalsetmessages: + msg357716
2019-12-02 16:16:17miss-islingtonsetpull_requests: + pull_request16920
2019-12-02 16:16:10miss-islingtonsetpull_requests: + pull_request16919
2019-12-02 16:15:46miss-islingtonsetnosy: + miss-islington
messages: + msg357715
2019-12-02 15:54:09torsavasetkeywords: + patch
stage: patch review
pull_requests: + pull_request16917
2019-12-02 15:32:21christian.heimessetmessages: + msg357712
2019-12-02 13:55:03pablogsalsetpriority: normal -> critical
nosy: + pablogsal
messages: + msg357711

2019-12-02 13:42:23vstinnersetmessages: + msg357710
title: test_ssl: test_min_max_version() fails on AMD64 FreeBSD Shared 3.x -> test_ssl: test_min_max_version() fails on FreeBSD and Fedora
2019-12-02 13:41:41vstinnerlinkissue38954 superseder
2019-12-02 13:28:42vstinnersetmessages: + msg357707
2019-12-02 13:10:55vstinnersetmessages: + msg357706
2019-12-02 13:07:32vstinnersetmessages: + msg357705
2019-11-18 11:01:21vstinnersetmessages: + msg356857
2019-11-15 16:55:11vstinnersetmessages: + msg356692
2019-11-15 16:54:20vstinnercreate