Example failures

https://buildbot.python.org/all/#/builders/117
https://buildbot.python.org/all/#/builders/106

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)
----------------------------------------------------------------------
Ran 105 tests in 2.477s

Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
Got an error:
[SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1055)
test_local_bad_hostname (test.test_httplib.HTTPSTest) ...  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
test_local_good_hostname (test.test_httplib.HTTPSTest) ...  server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] code 404, message File not found
 server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
   [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
stopping HTTPS server
joining HTTPS thread
ok
Got an error:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS server
joining HTTPS thread
ok

Multiple SSL failures, also old commits that previously succeeded fail now. This seems something in the buildbot itself. Gregory, do you know if something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l worker?

FYI - the name of this bot is misleading. It is now Debian testing as of 18
hours ago instead of obsolete Ubuntu 14.04.  I finally upgraded it.

Opens version says 1.1.1a.

--
blame half the typos on my phone.

On Wed, Feb 6, 2019, 5:45 PM Pablo Galindo Salgado <report@bugs.python.org
wrote:

>
> New submission from Pablo Galindo Salgado <pablogsal@gmail.com>:
>
> Example failures
>
> https://buildbot.python.org/all/#/builders/117
> https://buildbot.python.org/all/#/builders/106
>
> ======================================================================
> ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py",
> line 1629, in test_networked_good_cert
>     h.request('GET', '/')
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1229, in request
>     self._send_request(method, url, body, headers, encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1275, in _send_request
>     self.endheaders(body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1224, in endheaders
>     self._send_output(message_body, encode_chunked=encode_chunked)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1016, in _send_output
>     self.send(msg)
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 956, in send
>     self.connect()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py",
> line 1391, in connect
>     self.sock = self._context.wrap_socket(self.sock,
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 405, in wrap_socket
>     return self.sslsocket_class._create(
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 853, in _create
>     self.do_handshake()
>   File
> "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py",
> line 1117, in do_handshake
>     self._sslobj.do_handshake()
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate
> verify failed: EE certificate key too weak (_ssl.c:1055)
> ----------------------------------------------------------------------
> Ran 105 tests in 2.477s
>
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> Got an error:
> [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate
> (_ssl.c:1055)
> test_local_bad_hostname (test.test_httplib.HTTPSTest) ...  server
> (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 41921):41921 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
> stopping HTTPS server
> joining HTTPS thread
> ok
> test_local_good_hostname (test.test_httplib.HTTPSTest) ...  server
> (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)):
>    [06/Feb/2019 06:22:07] code 404, message File not found
>  server (('127.0.0.1', 38877):38877 ('TLS_AES_256_GCM_SHA384', 'TLSv1.3',
> 256)):
>    [06/Feb/2019 06:22:07] "GET /nonexistent HTTP/1.1" 404 -
> stopping HTTPS server
> joining HTTPS thread
> ok
> Got an error:
> [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1055)
> test_local_unknown_cert (test.test_httplib.HTTPSTest) ... stopping HTTPS
> server
> joining HTTPS thread
> ok
>
> Multiple SSL failures, also old commits that previously succeeded fail
> now. This seems something in the buildbot itself. Gregory, do you know if
> something SLL related was upgraded/modify in the gps-ubuntu-exynos5-armv7l
> worker?
>
> ----------
> components: Tests
> messages: 334996
> nosy: gregory.p.smith, pablogsal
> priority: normal
> severity: normal
> status: open
> title: test_httplib test_nntplib test_ssl fail on ARMv7 Ubuntu 3.7 and
> ARMv7 Ubuntu 3.x buildbots
> versions: Python 3.7, Python 3.8
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue35925>
> _______________________________________
>

I had emailed Christian around the same time you filed this.

"""
The problem likely not related to your hardware. I guess it's caused by
tightened crypto polices. OpenSSL 1.1.1 has disabled some weak crypto.
Some platforms like Debian and RHEL require even larger key sizes or
have disable some algorithms. Does the test also fail with the env var
OPENSSL_CONF set to a non-existing path?
""" - christian.heimes

testing that theory... setting OPENSSL_CONF=/invalid-path does indeed "fix" (work around) the failures.  Presumably by relaxing the default system constraints.

I could have that env var set for this buildbot and eliminate the failure.  But do we _want_ to do that?  Anyone who compiles CPython and tries to run the test suite on a modern system with such an OpenSSL configuration is going to see similar failures and likely come to us first asking about them.

It seems like we'd be better off adjusting our test suite to work around the constraints or disable them only for the duration of a test intentionally violating them?

Does test_ssl pass on the master branch?

Not on this debian buster bot.  look back a couple comments, there is a workaround.  it seems to be an OpenSSL configuration issue / test expectations issue.

I think we should ultimately get our test suite so that it passes in default OS distro OpenSSL configs.

That could mean any of an altered environment for some tests, or skipping some tests in such an environment, or changing some tests to fit within modern OpenSSL desired ciphersuite/protocol setting constrains constraints - or a mix of all three.

release managers are free to defer this blocker.  i'm just marking it as such for the purposes of making sure it is a conscious decision.

The problem is more likely with our test suite vs the environment than it is with CPython itself.

FWIW I've just manually confirmed that running Python 2.7's test_ssl with OPENSSL_CONF=/invalid-path set passes on the debian buster buildbot host.

I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.

Getting those failures on RHEL8 as well, which can be worked around by setting the env OPENSSL_CONF=/non-existing-file


======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2370, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)

======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2444, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2103, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/root/cpython/_install/lib/python2.7/test/test_ssl.py", line 2031, in server_params_test
    s.connect((HOST, server.port))
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 864, in connect
    self._real_connect(addr, False)
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 855, in _real_connect
    self.do_handshake()
  File "/root/cpython/_install/lib/python2.7/ssl.py", line 828, in do_handshake
    self._sslobj.do_handshake()
SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:727)

I wrote a fix for bpo-36037 "test_ssl fails on RHEL8 strict OpenSSL configuration" which should fix test_ssl on Debian as well, but my change doesn't apply to Python 2.7 nor 3.6 since these Python versions lack SSLContext.minimum_version attribute (introduced in Python 3.7).

https://docs.python.org/dev/library/ssl.html#ssl.SSLContext.minimum_version

For Python 2.7 and 3.6, "export OPENSSL_CONF=/non-existing-file" is a workaround.

Benjamin:
> I agree that we need to be more resistant to system configuration, but it doesn't seem worth holding 2.7 up for.

My fix requires SSLContext.minimum_version, but I'm not sure that it's ok to backport the attribute to Python 2.7 since Python 3.6 doesn't have it. IMHO "export OPENSSL_CONF=/non-existing-file" workaround is acceptable.

It's okay with me if you want to backport minimum_version (and I suppose maximum_version).

SSLContext.minimum_version is added here on the master branch:

https://github.com/python/cpython/commit/698dde16f60729d9e3f53c23a4ddb8e5ffe818bf

But I'd be also reluctant to partially backport a new feature to fix the test suite.

After my change:

commit 3ef6344ee53f59ee86831ec36ed2c6f93a56229d
Author: Victor Stinner <vstinner@redhat.com>
Date:   Tue Feb 19 18:06:03 2019 +0100

    bpo-36037: Fix test_ssl for strict OpenSSL policy (GH-11940)

Two tests are still failing on the Debian buildbot worker:

ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)

We should use different servers or contact admins of these servers to update their TLS configuration and/or certificate.

bpo-36104 has been marked as a duplicate of this issue. Copy of Lukasz's msg336511:

The ARMv7 Ubuntu buildbot is consistently failing since build #2160:
https://buildbot.python.org/all/#/builders/106/builds/2160


This looks like a testing environment issue to me rather than a code issue. But I'd like it fixed either way before we get to 3.8.0 beta1 since this is a stable builder. Greg, you can ask Inadasan about whether his dict/OrderedDict changes might have any effect on this failure:
https://github.com/python/cpython/commit/c95404ff65dab1469dcd1dfec58ba54a8e7e7b3a

That was the only relevant change I observed between the working and the broken build.


The NNTP test failure looks like this:

======================================================================
ERROR: setUpClass (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_nntplib.py", line 295, in setUpClass
    cls.server = cls.NNTP_CLASS(cls.NNTP_HOST, timeout=TIMEOUT,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 1077, in __init__
    self.sock = _encrypt_on(self.sock, ssl_context, host)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/nntplib.py", line 292, in _encrypt_on
    return context.wrap_socket(sock, server_hostname=hostname)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1055)



The HTTP test failure looks like this:

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_httplib.py", line 1629, in test_networked_good_cert
    h.request('GET', '/')
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 956, in send
    self.connect()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/http/client.py", line 1391, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.x.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: EE certificate key too weak (_ssl.c:1055)

Lukasz: this issue is that Debian Buster uses a strict OpenSSL policy. I guess that external public server used by tests are incompatible with this strict policy.

This is still failing regularly - any progress? Do we need to skip tests?

While altering the environment to not use the system default openssl config is an option to make this green again today very easily.  That'd "solve" the red bot problem and nothing else. :/

Doing that just kicks the can down the road as all of us Linux users are going to face this problem when we start using modern OS distros to build and test CPython.

A skipped test is an ignored test.

Ideally I'd like to see the tests updated to comply with modern higher security openssl config constraints.

PR coming

New changeset 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90 by Gregory P. Smith in branch 'master':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
https://github.com/python/cpython/commit/2cc0223f43a1ffd59c887a73e2b0ce5202f3be90

New changeset ffa29b5aca1aaeae46af2582c401ef0ed20d4153 by Miss Islington (bot) in branch '3.7':
bpo-35925: Skip SSL tests that fail due to weak external certs. (GH-13124)
https://github.com/python/cpython/commit/ffa29b5aca1aaeae46af2582c401ef0ed20d4153

The merged PR basically skips the specific failing unit test cases of the ssl key strength check error is detected during these network tests.  It should probably be backported into 3.6 and 2.7 to ease maintenance and trust of the buildbots on those.

Only people running regrtest -u all or at least -u networking to enable the live network connectivity tests would run into this when building their own CPython.

I'm still seeing the issue on https://github.com/python/cpython/pull/12255 (freshly rebased to master to have 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90.

On this build: https://dev.azure.com/Python/cpython/_build/results?buildId=42065

======================================================================
ERROR: test_networked_good_cert (test.test_httplib.HTTPSTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vsts/work/1/s/Lib/test/test_httplib.py", line 1632, in test_networked_good_cert
    h.request('GET', '/')
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1221, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1267, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1216, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1004, in _send_output
    self.send(msg)
  File "/home/vsts/work/1/s/Lib/http/client.py", line 944, in send
    self.connect()
  File "/home/vsts/work/1/s/Lib/http/client.py", line 1383, in connect
    self.sock = self._context.wrap_socket(self.sock,
  File "/home/vsts/work/1/s/Lib/ssl.py", line 405, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/vsts/work/1/s/Lib/ssl.py", line 853, in _create
    self.do_handshake()
  File "/home/vsts/work/1/s/Lib/ssl.py", line 1117, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1055)

which does not looks covered by 2cc0223f43a1ffd59c887a73e2b0ce5202f3be90 which only checks for key too weak.

thats https://bugs.python.org/issue36816 (separate issue as our infrastructure is fixed to have a modern certificate).  PR pending automerge post-CI.

👍

In our 3.6 tree the test_ssl failure is now:

======================================================================
ERROR: test_protocol_sslv23 (test.test_ssl.ThreadedTests)
Connecting to an SSLv23 server with various client options
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2633, in test_protocol_sslv23
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1, 'TLSv1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)
======================================================================
ERROR: test_protocol_tlsv1_1 (test.test_ssl.ThreadedTests)
Connecting to a TLSv1.1 server with various client options.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2707, in test_protocol_tlsv1_1
    try_protocol_combo(ssl.PROTOCOL_SSLv23, ssl.PROTOCOL_TLSv1_1, 'TLSv1.1')
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2323, in try_protocol_combo
    chatty=False, connectionchatty=False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/test/test_ssl.py", line 2248, in server_params_test
    s.connect((HOST, server.port))
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1109, in connect
    self._real_connect(addr, False)
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1100, in _real_connect
    self.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 1077, in do_handshake
    self._sslobj.do_handshake()
  File "/ssd/buildbot/buildarea/3.6.gps-ubuntu-exynos5-armv7l/build/Lib/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:852)

(same on 2.7)

New changeset 7346a16ed584fd1e85359154820d286370b68648 by Gregory P. Smith in branch '2.7':
[2.7] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13253)
https://github.com/python/cpython/commit/7346a16ed584fd1e85359154820d286370b68648

3.6 (and 3.5 if larry wants) are the only remaining trees to apply this to, assigning to the 3.6 RM.

New changeset 8ab624b17ba656e9af5a79be6af0cf2911a111ba by Ned Deily (Gregory P. Smith) in branch '3.6':
[3.6] bpo-35925: Skip SSL tests that fail due to weak external certs or old TLS (GH-13124) (GH-13252)
https://github.com/python/cpython/commit/8ab624b17ba656e9af5a79be6af0cf2911a111ba