test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106

pablogsal · 2019-02-07T01:45:26Z

BPO	35925
Nosy	@gpshead, @vstinner, @tiran, @benjaminp, @ned-deily, @ambv, @zooba, @JulienPalard, @stratakis, @pablogsal, @miss-islington
PRs	bpo-35925: Skip SSL tests that fail due to weak external certs. #13124 [3.7] bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124) #13139 [3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) #13252 [2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) #13253

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-05-29.02:12:18.543>
created_at = <Date 2019-02-07.01:45:26.293>
labels = ['3.7', '3.8', 'type-bug', 'tests']
title = 'test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)'
updated_at = <Date 2019-05-29.02:12:18.530>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2019-05-29.02:12:18.530>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2019-05-29.02:12:18.543>
closer = 'ned.deily'
components = ['Tests']
creation = <Date 2019-02-07.01:45:26.293>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 35925
keywords = ['patch']
message_count = 29.0
messages = ['334996', '335001', '335047', '335630', '335639', '335640', '335642', '335709', '335949', '335968', '336038', '336056', '336212', '336520', '336521', '340086', '340090', '341584', '341650', '341678', '341679', '341902', '341903', '341913', '342172', '342173', '342382', '342384', '343855']
nosy_count = 11.0
nosy_names = ['gregory.p.smith', 'vstinner', 'christian.heimes', 'benjamin.peterson', 'ned.deily', 'lukasz.langa', 'steve.dower', 'mdk', 'cstratak', 'pablogsal', 'miss-islington']
pr_nums = ['13124', '13139', '13252', '13253']
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35925'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8']

pablogsal · 2019-02-07T01:45:26Z

Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ... server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail now. This seems something in the buildbot itself. Gregory, do you know if something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l worker?

gpshead · 2019-02-07T04:16:50Z

FYI - the name of this bot is misleading. It is now Debian testing as of 18
hours ago instead of obsolete Ubuntu 14.04. I finally upgraded it.

Opens version says 1.1.1a.

--
blame half the typos on my phone.

On Wed, Feb 6, 2019, 5:45 PM Pablo Galindo Salgado <report@bugs.python.org
wrote:

New submission from Pablo Galindo Salgado <pablogsal@gmail.com>:

Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py",
> line 1629, in test_networked_good_cert
>     h.request('GET', '/')
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1229, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1275, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1224, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1016, in _send_output
>     self.send(msg)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 956, in send
>     self.connect()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1391, in connect
>     self.sock = self._context.wrap_socket(self.sock,
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 405, in wrap_socket
>     return self.sslsocket_class._create(
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 853, in _create
>     self.do_handshake()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 1117, in do_handshake
>     self._sslobj.do_handshake()
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
> verify failed: EE certificate key too weak (_ssl.c:1055)
>

Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
(_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ... server
(('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ... server
(('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
[06/Feb/2019 06:22:07] code 404, message File not found
server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
256)):
[06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS
server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail
now. This seems something in the buildbot itself. Gregory, do you know if
something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l
worker?

----------
components: Tests
messages: 334996
nosy: gregory.p.smith, pablogsal
priority: normal
severity: normal
status: open
title: test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and
ARMv7 Ubuntu 3.x buildbots
versions: Python 3.7, Python 3.8

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue35925\>

gpshead · 2019-02-07T23:30:30Z

I had emailed Christian around the same time you filed this.

"""
The problem likely not related to your hardware. I guess it's caused by
tightened crypto polices. OpenSSL 1.1.1 has disabled some weak crypto.
Some platforms like Debian and RHEL require even larger key sizes or
have disable some algorithms. Does the test also fail with the env var
OPENSSL_CONF set to a non-existing path?
""" - christian.heimes

testing that theory... setting OPENSSL_CONF=/invalid-path does indeed "fix" (work around) the failures. Presumably by relaxing the default system constraints.

I could have that env var set for this buildbot and eliminate the failure. But do we _want_ to do that? Anyone who compiles CPython and tries to run the test suite on a modern system with such an OpenSSL configuration is going to see similar failures and likely come to us first asking about them.

It seems like we'd be better off adjusting our test suite to work around the constraints or disable them only for the duration of a test intentionally violating them?

vstinner · 2019-02-15T18:00:30Z

Does test_ssl pass on the master branch?

gpshead · 2019-02-15T19:10:41Z

Not on this debian buster bot. look back a couple comments, there is a workaround. it seems to be an OpenSSL configuration issue / test expectations issue.

I think we should ultimately get our test suite so that it passes in default OS distro OpenSSL configs.

That could mean any of an altered environment for some tests, or skipping some tests in such an environment, or changing some tests to fit within modern OpenSSL desired ciphersuite/protocol setting constrains constraints - or a mix of all three.

gpshead · 2019-02-15T19:18:54Z

release managers are free to defer this blocker. i'm just marking it as such for the purposes of making sure it is a conscious decision.

The problem is more likely with our test suite vs the environment than it is with CPython itself.

gpshead · 2019-02-15T19:22:35Z

FWIW I've just manually confirmed that running Python 2.7's test_ssl with OPENSSL_CONF=/invalid-path set passes on the debian buster buildbot host.

benjaminp · 2019-02-16T18:44:12Z

I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.

stratakis · 2019-02-19T14:17:24Z

Getting those failures on RHEL8 as well, which can be worked around by setting the env OPENSSL_CONF=/non-existing-file

======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2370, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)

======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2444, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)

vstinner · 2019-02-19T16:28:24Z

I wrote a fix for bpo-36037 "test_ssl fails on RHEL8 strict OpenSSL configuration" which should fix test_ssl on Debian as well, but my change doesn't apply to Python 2.7 nor 3.6 since these Python versions lack SSLContext.minimum_version attribute (introduced in Python 3.7).

https://docs.python.org/dev/library/ssl.html#ssl.SSLContext.minimum_version

For Python 2.7 and 3.6, "export OPENSSL_CONF=/non-existing-file" is a workaround.

Benjamin:

I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.

My fix requires SSLContext.minimum_version, but I'm not sure that it's ok to backport the attribute to Python 2.7 since Python 3.6 doesn't have it. IMHO "export OPENSSL_CONF=/non-existing-file" workaround is acceptable.

benjaminp · 2019-02-20T05:06:06Z

It's okay with me if you want to backport minimum_version (and I suppose maximum_version).

stratakis · 2019-02-20T10:30:08Z

SSLContext.minimum_version is added here on the master branch:

698dde1

But I'd be also reluctant to partially backport a new feature to fix the test suite.

vstinner · 2019-02-21T12:09:22Z

After my change:

commit 3ef6344
Author: Victor Stinner <vstinner@redhat.com>
Date: Tue Feb 19 18:06:03 2019 +0100

bpo-36037: Fix test_ssl for strict OpenSSL policy (GH-11940)

Two tests are still failing on the Debian buildbot worker:

ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)

We should use different servers or contact admins of these servers to update their TLS configuration and/or certificate.

vstinner · 2019-02-25T14:06:18Z

bpo-36104 has been marked as a duplicate of this issue. Copy of Lukasz's msg336511:

The ARMv7 Ubuntu buildbot is consistently failing since build bpo-2160:
https://buildbot.python.org/all/#/builders/106/builds/2160

This looks like a testing environment issue to me rather than a code issue. But I'd like it fixed either way before we get to 3.8.0 beta1 since this is a stable builder. Greg, you can ask Inadasan about whether his dict/OrderedDict changes might have any effect on this failure:
c95404f

That was the only relevant change I observed between the working and the broken build.

The NNTP test failure looks like this:

======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_nntplib.py", line 295, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, timeout=TIMEOUT,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 1077, in __init__
    self.sock = _encrypt_on(self.sock, ssl_context, host)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 292, in _encrypt_on
    return context.wrap_socket(sock, server_hostname=hostname)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)

The HTTP test failure looks like this:

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

vstinner · 2019-02-25T14:07:13Z

Lukasz: this issue is that Debian Buster uses a strict OpenSSL policy. I guess that external public server used by tests are incompatible with this strict policy.

zooba · 2019-04-12T17:23:05Z

This is still failing regularly - any progress? Do we need to skip tests?

gpshead · 2019-04-12T17:53:46Z

While altering the environment to not use the system default openssl config is an option to make this green again today very easily. That'd "solve" the red bot problem and nothing else. :/

Doing that just kicks the can down the road as all of us Linux users are going to face this problem when we start using modern OS distros to build and test CPython.

A skipped test is an ignored test.

Ideally I'd like to see the tests updated to comply with modern higher security openssl config constraints.

gpshead · 2019-05-06T18:23:02Z

PR coming

gpshead · 2019-05-06T21:54:29Z

New changeset 2cc0223 by Gregory P. Smith in branch 'master':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
2cc0223

miss-islington · 2019-05-07T03:51:32Z

New changeset ffa29b5 by Miss Islington (bot) in branch '3.7':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
ffa29b5

gpshead · 2019-05-07T03:56:10Z

The merged PR basically skips the specific failing unit test cases of the ssl key strength check error is detected during these network tests. It should probably be backported into 3.6 and 2.7 to ease maintenance and trust of the buildbots on those.

Only people running regrtest -u all or at least -u networking to enable the live network connectivity tests would run into this when building their own CPython.

JulienPalard · 2019-05-08T16:27:00Z

I'm still seeing the issue on #12255 (freshly rebased to master to have 2cc0223.

On this build: https://dev.azure.com/Python/cpython/_build/results?buildId=42065

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/vsts/work/1/s/Lib/test/test_httplib.py", line 1632, in test_networked_good_cert
    h.request('GET', '/')
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1221, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1267, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1216, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1004, in _send_output
    self.send(msg)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 944, in send
    self.connect()
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1383, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/home/vsts/work/1/s/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/vsts/work/1/s/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/home/vsts/work/1/s/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1055)

which does not looks covered by 2cc0223 which only checks for key too weak.

gpshead · 2019-05-08T16:29:47Z

thats https://bugs.python.org/issue36816 (separate issue as our infrastructure is fixed to have a modern certificate). PR pending automerge post-CI.

JulienPalard · 2019-05-08T17:12:40Z

👍

gpshead · 2019-05-11T07:10:01Z

In our 3.6 tree the test_ssl failure is now:

======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2633, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)

======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2707, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)

gpshead · 2019-05-11T07:11:46Z

(same on 2.7)

gpshead · 2019-05-13T20:16:46Z

New changeset 7346a16 by Gregory P. Smith in branch '2.7':
[2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13253)
7346a16

gpshead · 2019-05-13T20:30:37Z

3.6 (and 3.5 if larry wants) are the only remaining trees to apply this to, assigning to the 3.6 RM.

ned-deily · 2019-05-29T02:08:32Z

New changeset 8ab624b by Ned Deily (Gregory P. Smith) in branch '3.6':
[3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13252)
8ab624b

pablogsal added 3.7 (EOL) end of life 3.8 only security fixes tests Tests in the Lib/test dir labels Feb 7, 2019

gpshead changed the title ~~test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and ARMv7 Ubuntu 3.x buildbots~~ test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots Feb 7, 2019

gpshead changed the title ~~test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots~~ test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot Feb 7, 2019

gpshead added release-blocker type-bug An unexpected behavior, bug, or error labels Feb 15, 2019

gpshead changed the title ~~test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot~~ test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) Feb 15, 2019

benjaminp removed the release-blocker label Feb 16, 2019

gpshead self-assigned this May 6, 2019

gpshead removed 3.7 (EOL) end of life 3.8 only security fixes labels May 7, 2019

gpshead assigned ned-deily and unassigned gpshead May 13, 2019

ned-deily added 3.7 (EOL) end of life 3.8 only security fixes labels May 29, 2019

ned-deily closed this as completed May 29, 2019

ned-deily removed their assignment May 29, 2019

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106

test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106

pablogsal commented Feb 7, 2019

pablogsal commented Feb 7, 2019

gpshead commented Feb 7, 2019

gpshead commented Feb 7, 2019

vstinner commented Feb 15, 2019

gpshead commented Feb 15, 2019

gpshead commented Feb 15, 2019

gpshead commented Feb 15, 2019

benjaminp commented Feb 16, 2019

stratakis mannequin commented Feb 19, 2019

vstinner commented Feb 19, 2019

benjaminp commented Feb 20, 2019

stratakis mannequin commented Feb 20, 2019

vstinner commented Feb 21, 2019

vstinner commented Feb 25, 2019

vstinner commented Feb 25, 2019

zooba commented Apr 12, 2019

gpshead commented Apr 12, 2019

gpshead commented May 6, 2019

gpshead commented May 6, 2019

miss-islington commented May 7, 2019

gpshead commented May 7, 2019

JulienPalard commented May 8, 2019

gpshead commented May 8, 2019

JulienPalard commented May 8, 2019

gpshead commented May 11, 2019

gpshead commented May 11, 2019

gpshead commented May 13, 2019

gpshead commented May 13, 2019

ned-deily commented May 29, 2019

test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106

test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a) #80106

Comments

pablogsal commented Feb 7, 2019

pablogsal commented Feb 7, 2019

gpshead commented Feb 7, 2019

gpshead commented Feb 7, 2019

vstinner commented Feb 15, 2019

gpshead commented Feb 15, 2019

gpshead commented Feb 15, 2019

gpshead commented Feb 15, 2019

benjaminp commented Feb 16, 2019

stratakis mannequin commented Feb 19, 2019

vstinner commented Feb 19, 2019

benjaminp commented Feb 20, 2019

stratakis mannequin commented Feb 20, 2019

vstinner commented Feb 21, 2019

vstinner commented Feb 25, 2019

vstinner commented Feb 25, 2019

zooba commented Apr 12, 2019

gpshead commented Apr 12, 2019

gpshead commented May 6, 2019

gpshead commented May 6, 2019

miss-islington commented May 7, 2019

gpshead commented May 7, 2019

JulienPalard commented May 8, 2019

gpshead commented May 8, 2019

JulienPalard commented May 8, 2019

gpshead commented May 11, 2019

gpshead commented May 11, 2019

gpshead commented May 13, 2019

gpshead commented May 13, 2019

ned-deily commented May 29, 2019