classification
Title: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)
Type: behavior Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7, Python 3.6, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, christian.heimes, cstratak, gregory.p.smith, lukasz.langa, mdk, miss-islington, ned.deily, pablogsal, steve.dower, vstinner
Priority: high Keywords: patch

Created on 2019-02-07 01:45 by pablogsal, last changed 2019-05-29 02:12 by ned.deily. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13124 merged gregory.p.smith, 2019-05-06 18:27
PR 13139 merged miss-islington, 2019-05-06 21:54
PR 13252 merged gregory.p.smith, 2019-05-11 18:44
PR 13253 merged gregory.p.smith, 2019-05-11 20:44
Messages (29)
msg334996 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-02-07 01:45
Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)
----------------------------------------------------------------------
Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ...  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ...  server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail now. This seems something in the buildbot itself. Gregory, do you know if something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l worker?
msg335001 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-02-07 04:16
FYI - the name of this bot is misleading. It is now Debian testing as of 18
hours ago instead of obsolete Ubuntu 14.04.  I finally upgraded it.

Opens version says 1.1.1a.

--
blame half the typos on my phone.

On Wed, Feb 6, 2019, 5:45 PM Pablo Galindo Salgado <report@bugs.python.org
wrote:

>
> New submission from Pablo Galindo Salgado <pablogsal@gmail.com>:
>
> Example failures
>
> https://buildbot.python.org/all/#/builders/117
> https://buildbot.python.org/all/#/builders/106
>
> ======================================================================
> ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py",
> line 1629, in test_networked_good_cert
>     h.request('GET', '/')
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1229, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1275, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1224, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1016, in _send_output
>     self.send(msg)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 956, in send
>     self.connect()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1391, in connect
>     self.sock = self._context.wrap_socket(self.sock,
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 405, in wrap_socket
>     return self.sslsocket_class._create(
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 853, in _create
>     self.do_handshake()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 1117, in do_handshake
>     self._sslobj.do_handshake()
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
> verify failed: EE certificate key too weak (_ssl.c:1055)
> ----------------------------------------------------------------------
> Ran 105 tests in 2.477s
>
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> test_local_bad_hostname (test.test_httplib.HTTPSTest) ...  server
> (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
> stopping HTTPS server
> joining HTTPS thread
> ok
> test_local_good_hostname (test.test_httplib.HTTPSTest) ...  server
> (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
> stopping HTTPS server
> joining HTTPS thread
> ok
> Got an error:
> [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
> test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS
> server
> joining HTTPS thread
> ok
>
> Multiple SSL failures, also old commits that previously succeeded fail
> now. This seems something in the buildbot itself. Gregory, do you know if
> something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l
> worker?
>
> ----------
> components: Tests
> messages: 334996
> nosy: gregory.p.smith, pablogsal
> priority: normal
> severity: normal
> status: open
> title: test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and
> ARMv7 Ubuntu 3.x buildbots
> versions: Python 3.7, Python 3.8
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35925>
> _______________________________________
>
msg335047 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-02-07 23:30
I had emailed Christian around the same time you filed this.

"""
The problem likely not related to your hardware. I guess it's caused by
tightened crypto polices. OpenSSL 1.1.1 has disabled some weak crypto.
Some platforms like Debian and RHEL require even larger key sizes or
have disable some algorithms. Does the test also fail with the env var
OPENSSL_CONF set to a non-existing path?
""" - christian.heimes

testing that theory... setting OPENSSL_CONF=/invalid-path does indeed "fix" (work around) the failures.  Presumably by relaxing the default system constraints.

I could have that env var set for this buildbot and eliminate the failure.  But do we _want_ to do that?  Anyone who compiles CPython and tries to run the test suite on a modern system with such an OpenSSL configuration is going to see similar failures and likely come to us first asking about them.

It seems like we'd be better off adjusting our test suite to work around the constraints or disable them only for the duration of a test intentionally violating them?
msg335630 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-15 18:00
Does test_ssl pass on the master branch?
msg335639 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-02-15 19:10
Not on this debian buster bot.  look back a couple comments, there is a workaround.  it seems to be an OpenSSL configuration issue / test expectations issue.

I think we should ultimately get our test suite so that it passes in default OS distro OpenSSL configs.

That could mean any of an altered environment for some tests, or skipping some tests in such an environment, or changing some tests to fit within modern OpenSSL desired ciphersuite/protocol setting constrains constraints - or a mix of all three.
msg335640 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-02-15 19:18
release managers are free to defer this blocker.  i'm just marking it as such for the purposes of making sure it is a conscious decision.

The problem is more likely with our test suite vs the environment than it is with CPython itself.
msg335642 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-02-15 19:22
FWIW I've just manually confirmed that running Python 2.7's test_ssl with OPENSSL_CONF=/invalid-path set passes on the debian buster buildbot host.
msg335709 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-02-16 18:44
I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.
msg335949 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-02-19 14:17
Getting those failures on RHEL8 as well, which can be worked around by setting the env OPENSSL_CONF=/non-existing-file


======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2370, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)

======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2444, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)
msg335968 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-19 16:28
I wrote a fix for bpo-36037 "test_ssl fails on RHEL8 strict OpenSSL configuration" which should fix test_ssl on Debian as well, but my change doesn't apply to Python 2.7 nor 3.6 since these Python versions lack SSLContext.minimum_version attribute (introduced in Python 3.7).

https://docs.python.org/dev/library/ssl.html#ssl.SSLContext.minimum_version

For Python 2.7 and 3.6, "export OPENSSL_CONF=/non-existing-file" is a workaround.

Benjamin:
> I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.

My fix requires SSLContext.minimum_version, but I'm not sure that it's ok to backport the attribute to Python 2.7 since Python 3.6 doesn't have it. IMHO "export OPENSSL_CONF=/non-existing-file" workaround is acceptable.
msg336038 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2019-02-20 05:06
It's okay with me if you want to backport minimum_version (and I suppose maximum_version).
msg336056 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2019-02-20 10:30
SSLContext.minimum_version is added here on the master branch:

https://github.com/python/cpython/commit/698dde16f60729d9e3f53c23a4ddb8e5ffe818bf

But I'd be also reluctant to partially backport a new feature to fix the test suite.
msg336212 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-21 12:09
After my change:

commit 3ef6344ee53f59ee86831ec36ed2c6f93a56229d
Author: Victor Stinner <vstinner@redhat.com>
Date:   Tue Feb 19 18:06:03 2019 +0100

    bpo-36037: Fix test_ssl for strict OpenSSL policy (GH-11940)

Two tests are still failing on the Debian buildbot worker:

ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)

We should use different servers or contact admins of these servers to update their TLS configuration and/or certificate.
msg336520 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 14:06
bpo-36104 has been marked as a duplicate of this issue. Copy of Lukasz's msg336511:

The ARMv7 Ubuntu buildbot is consistently failing since build #2160:
https://buildbot.python.org/all/#/builders/106/builds/2160


This looks like a testing environment issue to me rather than a code issue. But I'd like it fixed either way before we get to 3.8.0 beta1 since this is a stable builder. Greg, you can ask Inadasan about whether his dict/OrderedDict changes might have any effect on this failure:
https://github.com/python/cpython/commit/c95404ff65dab1469dcd1dfec58ba54a8e7e7b3a

That was the only relevant change I observed between the working and the broken build.


The NNTP test failure looks like this:

======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_nntplib.py", line 295, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, timeout=TIMEOUT,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 1077, in __init__
    self.sock = _encrypt_on(self.sock, ssl_context, host)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 292, in _encrypt_on
    return context.wrap_socket(sock, server_hostname=hostname)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)



The HTTP test failure looks like this:

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)
msg336521 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-02-25 14:07
Lukasz: this issue is that Debian Buster uses a strict OpenSSL policy. I guess that external public server used by tests are incompatible with this strict policy.
msg340086 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-04-12 17:23
This is still failing regularly - any progress? Do we need to skip tests?
msg340090 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-04-12 17:53
While altering the environment to not use the system default openssl config is an option to make this green again today very easily.  That'd "solve" the red bot problem and nothing else. :/

Doing that just kicks the can down the road as all of us Linux users are going to face this problem when we start using modern OS distros to build and test CPython.

A skipped test is an ignored test.

Ideally I'd like to see the tests updated to comply with modern higher security openssl config constraints.
msg341584 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-06 18:23
PR coming
msg341650 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-06 21:54
New changeset 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90 by Gregory P. Smith in branch 'master':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
https://github.com/python/cpython/commit/2cc0223f43a1ffd59c887a73e2b0ce5202f3be90
msg341678 - (view) Author: miss-islington (miss-islington) Date: 2019-05-07 03:51
New changeset ffa29b5aca1aaeae46af2582c401ef0ed20d4153 by Miss Islington (bot) in branch '3.7':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
https://github.com/python/cpython/commit/ffa29b5aca1aaeae46af2582c401ef0ed20d4153
msg341679 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-07 03:56
The merged PR basically skips the specific failing unit test cases of the ssl key strength check error is detected during these network tests.  It should probably be backported into 3.6 and 2.7 to ease maintenance and trust of the buildbots on those.

Only people running regrtest -u all or at least -u networking to enable the live network connectivity tests would run into this when building their own CPython.
msg341902 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2019-05-08 16:27
I'm still seeing the issue on https://github.com/python/cpython/pull/12255 (freshly rebased to master to have 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90.

On this build: https://dev.azure.com/Python/cpython/_build/results?buildId=42065

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/Lib/test/test_httplib.py", line 1632, in test_networked_good_cert
    h.request('GET', '/')
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1221, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1267, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1216, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1004, in _send_output
    self.send(msg)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 944, in send
    self.connect()
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1383, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/home/vsts/work/1/s/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/vsts/work/1/s/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/home/vsts/work/1/s/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1055)

which does not looks covered by 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90 which only checks for key too weak.
msg341903 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-08 16:29
thats https://bugs.python.org/issue36816 (separate issue as our infrastructure is fixed to have a modern certificate).  PR pending automerge post-CI.
msg341913 - (view) Author: Julien Palard (mdk) * (Python committer) Date: 2019-05-08 17:12
👍
msg342172 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-11 07:10
In our 3.6 tree the test_ssl failure is now:

======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2633, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)
======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2707, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)
msg342173 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-11 07:11
(same on 2.7)
msg342382 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-13 20:16
New changeset 7346a16ed584fd1e85359154820d286370b68648 by Gregory P. Smith in branch '2.7':
[2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13253)
https://github.com/python/cpython/commit/7346a16ed584fd1e85359154820d286370b68648
msg342384 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2019-05-13 20:30
3.6 (and 3.5 if larry wants) are the only remaining trees to apply this to, assigning to the 3.6 RM.
msg343855 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2019-05-29 02:08
New changeset 8ab624b17ba656e9af5a79be6af0cf2911a111ba by Ned Deily (Gregory P. Smith) in branch '3.6':
[3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13252)
https://github.com/python/cpython/commit/8ab624b17ba656e9af5a79be6af0cf2911a111ba
History
Date User Action Args
2019-05-29 02:12:18ned.deilysetstatus: open -> closed
assignee: ned.deily ->
stage: patch review -> resolved
resolution: fixed
versions: + Python 2.7, Python 3.7, Python 3.8
2019-05-29 02:08:31ned.deilysetmessages: + msg343855
2019-05-13 20:30:37gregory.p.smithsetassignee: gregory.p.smith -> ned.deily
messages: + msg342384
versions: - Python 2.7
2019-05-13 20:16:46gregory.p.smithsetmessages: + msg342382
2019-05-11 20:44:38gregory.p.smithsetpull_requests: + pull_request13165
2019-05-11 18:44:00gregory.p.smithsetstage: backport needed -> patch review
pull_requests: + pull_request13163
2019-05-11 07:11:46gregory.p.smithsetmessages: + msg342173
2019-05-11 07:10:00gregory.p.smithsetmessages: + msg342172
2019-05-08 17:12:39mdksetmessages: + msg341913
2019-05-08 16:29:46gregory.p.smithsetmessages: + msg341903
2019-05-08 16:27:00mdksetnosy: + mdk
messages: + msg341902
2019-05-08 11:55:05yan12125setnosy: - yan12125
2019-05-08 03:46:19yan12125setnosy: + yan12125
2019-05-07 03:56:09gregory.p.smithsetstage: patch review -> backport needed
messages: + msg341679
versions: + Python 3.6, - Python 3.7, Python 3.8
2019-05-07 03:51:31miss-islingtonsetnosy: + miss-islington
messages: + msg341678
2019-05-06 21:54:28gregory.p.smithsetmessages: + msg341650
2019-05-06 21:54:18miss-islingtonsetpull_requests: + pull_request13052
2019-05-06 18:27:45gregory.p.smithsetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request13036
2019-05-06 18:23:01gregory.p.smithsetassignee: gregory.p.smith
messages: + msg341584
2019-04-12 17:53:46gregory.p.smithsetmessages: + msg340090
2019-04-12 17:23:05steve.dowersetnosy: + steve.dower
messages: + msg340086
2019-02-25 14:07:12vstinnersetnosy: + lukasz.langa
messages: + msg336521
2019-02-25 14:06:18vstinnersetmessages: + msg336520
2019-02-25 14:05:33vstinnerlinkissue36104 superseder
2019-02-21 12:09:22vstinnersetmessages: + msg336212
2019-02-20 10:30:08cstrataksetmessages: + msg336056
2019-02-20 05:06:06benjamin.petersonsetmessages: + msg336038
2019-02-19 16:28:24vstinnersetmessages: + msg335968
2019-02-19 14:17:24cstrataksetnosy: + cstratak
messages: + msg335949
2019-02-16 18:44:11benjamin.petersonsetpriority: release blocker -> high

messages: + msg335709
2019-02-15 19:22:35gregory.p.smithsetmessages: + msg335642
2019-02-15 19:18:54gregory.p.smithsetmessages: + msg335640
2019-02-15 19:17:44gregory.p.smithsettitle: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot -> test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster bot (OpenSSL 1.1.1a)
2019-02-15 19:17:27gregory.p.smithsetpriority: normal -> release blocker
nosy: + ned.deily, benjamin.peterson

type: behavior
stage: needs patch
2019-02-15 19:10:41gregory.p.smithsetmessages: + msg335639
versions: + Python 2.7
2019-02-15 18:00:30vstinnersetnosy: + vstinner
messages: + msg335630
2019-02-15 17:45:20gregory.p.smithlinkissue36005 superseder
2019-02-07 23:32:56gregory.p.smithsettitle: test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots -> test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster buildbot
2019-02-07 23:30:29gregory.p.smithsetmessages: + msg335047
title: test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and ARMv7 Ubuntu 3.x buildbots -> test_httplib test_nntplib test_ssl fail on ARMv7 buster/sid buildbots
2019-02-07 04:19:04gregory.p.smithsetnosy: + christian.heimes
2019-02-07 04:16:50gregory.p.smithsetmessages: + msg335001
2019-02-07 01:45:26pablogsalcreate