classification
Title: github CI blocked by the Ubuntu CI with an SSL error
Type: behavior Stage: resolved
Components: Build, SSL, Tests Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: brandtbucher, christian.heimes, gregory.p.smith, lukasz.langa, martonivan, nascheme, ned.deily, p-ganssle, vstinner, xnox
Priority: release blocker Keywords: patch

Created on 2021-03-03 04:29 by gregory.p.smith, last changed 2021-03-18 08:32 by christian.heimes. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 24715 merged brandtbucher, 2021-03-03 06:05
Messages (21)
msg387998 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-03 04:29
https://github.com/python/cpython/pull/20442/checks?check_run_id=2018900756

 ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1122)
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)
 ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1122)
 ssl.SSLError: [SSL] internal error (_ssl.c:1122)
 ssl.SSLError: [SSL] called a function you should not call (_ssl.c:1122)
 ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1122)
 ssl.SSLError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:1122)
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1122)
[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1122)

... and so on ...

This CI failure is preventing any PR from being merged.

Do we have a bad test certificate in the tree we need to update?
Or did some Ubuntu CI infrastructure just fall over and start rejecting a certificate it used to accept?

I believe this started around ~20210302T1500 UTC.
msg388000 - (view) Author: Brandt Bucher (brandtbucher) * (Python committer) Date: 2021-03-03 05:35
It seems that GitHub recently changed their "ubuntu-latest" image from Ubuntu 18.04 to Ubuntu 20.04.

A good temporary workaround would probably be to change this line:

https://github.com/python/cpython/blob/727a68b6e592eada5a65935de5c8428ef50e8741/.github/workflows/build.yml#L130

...to read "ubuntu-18.04".
msg388001 - (view) Author: Brandt Bucher (brandtbucher) * (Python committer) Date: 2021-03-03 05:48
I forgot to mention that I confirmed that the last passing test run used 18.04 (click "set up job" -> "Operating System" to see):

https://github.com/python/cpython/runs/2013210763?check_suite_focus=true

The next one, which started the current chain of failures, used 20.04:

https://github.com/python/cpython/runs/2018900756?check_suite_focus=true

I'm still not sure *why* this broke CI, though. Perhaps our cached OpenSSL is no longer valid?
msg388002 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2021-03-03 06:42
I think it may be related to bpo-41561.  There is a bug in the Ubuntu tracker as well:

https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878

I agree with the temporary fix to use "ubuntu-18.04" for CI testing.
msg388004 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 07:29
Thanks for the quick workaround!

The problem could be caused by a downstream patch in Ubuntu's OpenSSL version. Vanilla OpenSSL doesn't fail like that.
msg388006 - (view) Author: Dimitri John Ledkov (xnox) * Date: 2021-03-03 08:59
Ubuntu 20.04+ compile OpenSSL with default security level set to 2, and further customized security level 2 to prohibit TLS below v1.2 and DTLS below v1.2.

You can export custom openssl configuration that sets security level back to 1, which is compatible across any openssl series.

```
export OPENSSL_CONF=`pwd`/openssl.cnf
cat openssl.cnf
openssl_conf = default_conf
[default_conf]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
CipherString = DEFAULT@SECLEVEL=1
```

Or you can use native APIs to reset the security level to 1 in the test-suite. I.e. via the SSL_CTX_set_security_level api binding.

This is documented behaviour in Ubuntu manpages of OpenSSL and on Ubuntu Discourse

https://manpages.ubuntu.com/manpages/focal/en/man3/SSL_CTX_set_security_level.3ssl.html

https://discourse.ubuntu.com/t/default-to-tls-v1-2-in-all-tls-libraries-in-20-04-lts/12464/8

OpenSSL upstream for 3.0.0 series are refusing to bump minimum required protocol versions to prohibit out of the box old version of TLS and also don't have a standard way to disable this. Hence implementation is different in Debian, Ubuntu and Fedora. Debian's implementation is buggy with respect to DTLS and default openssl.cnf breaks 1.0.2x series libssl. And as far as I know Fedora implementation requires use of crypto-policies package which is quite advanced and not trivial to integrate in smaller environments.
msg388007 - (view) Author: Dimitri John Ledkov (xnox) * Date: 2021-03-03 09:00
BTW.

It would be advisable for Python3 to start enforcing security level 2, and prohibit DTLS v1.1 and lower by default too. By configuring openssl library on the host with setting security level, and/or setting min versions (if openssl on the host supports such api). Because allowing to use TLS v1.1 and lower out of the box is irresponsible.
msg388009 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 09:23
Dimitri, thanks for your feedback.

I'm very well aware of the crypto policy settings and security level settings. The problem is not the fact that Ubuntu sets a higher security level and disables insecure TLS versions. The problem is the way how Ubuntu has implemented the policy to enforce the crypto settings.

Other Linux distributions like Debian and Fedora also raise the security level and disable TLS 1.0 and 1.1. Python's test suite introspects OpenSSL settings and skips tests accordingly. test_ssl is passing fine on Debian testing (updated 15 minutes ago) and Fedora 33 with similar crypto policies. Since the tests are working fine on Debian, Fedora, RHEL/CentOS, vanilla OpenSSL, our OpenSSL builds on macOS and Windows, and other Linux distros, the issue is likely caused by a downstream discrepancy in Ubuntu.

# Python main branch on Fedora 33
$ ./python 
Python 3.10.0a5+ (heads/master:cd80f430daa, Feb 24 2021, 19:44:57) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version
<TLSVersion.TLSv1_2: 771>
>>> ctx.security_level
2

# Python main branch on Debian testing
$ ./python 
Python 3.10.0a6+ (heads/master:94894dd45e, Mar  3 2021, 09:11:22) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version
<TLSVersion.TLSv1_2: 771>
>>> ctx.security_level
2

$ ./python -m test test_ssl
0:00:00 load avg: 0.89 Run tests sequentially
0:00:00 load avg: 0.89 [1/1] test_ssl

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.6 sec
Tests result: SUCCESS

# dpkg -l openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-====================================================
ii  openssl        1.1.1j-1     amd64        Secure Sockets Layer toolkit - cryptographic utility
msg388010 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 09:27
> It would be advisable for Python3 to start enforcing security level 2, and prohibit DTLS v1.1 and lower by default too. By configuring openssl library on the host with setting security level, and/or setting min versions (if openssl on the host supports such api). Because allowing to use TLS v1.1 and lower out of the box is irresponsible.

We are going to change the default settings in our own OpenSSL builds together with https://www.python.org/dev/peps/pep-0644/ . For Linux distros we will rely on distro-wide crypto policies.
msg388011 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 09:58
I have backported the workaround to 3.7, 3.8, and 3.9. There was some issue with the backport bot and I didn't have time to investigate. PRs are:

https://github.com/python/cpython/pull/24716
https://github.com/python/cpython/pull/24717
https://github.com/python/cpython/pull/24718
msg388019 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 13:45
Downstream has asked me to file a separate bug for internal error during handshake. The problem is tracked at https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625 .
msg388031 - (view) Author: Brandt Bucher (brandtbucher) * (Python committer) Date: 2021-03-03 16:46
@ned.deily, I think the 3.7 backport needs RM approval (or something?):

https://github.com/python/cpython/pull/24716

The others branches are fine now... thanks, Christian!
msg388035 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-03 17:28
3.7 backport is now merged, too.
msg388043 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-03 18:51
How do I get a CI run on a PR to actually pickup this change?  clicking rerun failed.
msg388045 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2021-03-03 19:01
You may have to refresh the PR so that is uses the latest HEAD that includes the config change.
msg388047 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-03 19:19
Yeah, that's the annoying part. Users have to rebase all their PRs in order to make CI pass. It's going to be painful. :(
msg388048 - (view) Author: Brandt Bucher (brandtbucher) * (Python committer) Date: 2021-03-03 19:47
Closing and reopening may work, or pushing an empty commit. I know that's helped appease some GitHub CI weirdness in the past.
msg388052 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-03 20:08
yeah i figured it might require a rebase.  if anyone has the appropriate git command to do that to a branch, creating a 2-4 step CLI playbook for people to apply to pending PR branches would be useful.
msg388055 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2021-03-03 20:28
we may not need that: Closing and reopening the PR worked.  I suspect the github CI "rerun" button was "rerun in the exact same config".
msg388060 - (view) Author: Neil Schemenauer (nascheme) * (Python committer) Date: 2021-03-03 21:18
It seems it is enough to make a new commit.  The CI seems to re-base and re-run the PR.  At least, it worked on two of my PRs.
msg388999 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-03-18 08:32
CI is passing again.
History
Date User Action Args
2021-03-18 08:32:27christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg388999

stage: patch review -> resolved
2021-03-08 21:58:20vstinnersetnosy: + vstinner
2021-03-03 21:18:48naschemesetmessages: + msg388060
2021-03-03 20:28:02gregory.p.smithsetmessages: + msg388055
2021-03-03 20:08:51gregory.p.smithsetmessages: + msg388052
2021-03-03 19:47:44brandtbuchersetmessages: + msg388048
2021-03-03 19:19:28christian.heimessetmessages: + msg388047
2021-03-03 19:01:07ned.deilysetmessages: + msg388045
2021-03-03 18:51:37gregory.p.smithsetmessages: + msg388043
2021-03-03 17:28:10ned.deilysetmessages: + msg388035
2021-03-03 16:46:53brandtbuchersetmessages: + msg388031
2021-03-03 16:11:27p-gansslesetnosy: + p-ganssle
2021-03-03 13:45:41christian.heimessetmessages: + msg388019
2021-03-03 13:20:38martonivansetnosy: + martonivan
2021-03-03 09:58:42christian.heimessetmessages: + msg388011
2021-03-03 09:27:53christian.heimessetmessages: + msg388010
2021-03-03 09:23:41christian.heimessetmessages: + msg388009
2021-03-03 09:00:57xnoxsetmessages: + msg388007
2021-03-03 08:59:15xnoxsetnosy: + xnox
messages: + msg388006
2021-03-03 07:29:51christian.heimessetmessages: + msg388004
2021-03-03 06:42:16naschemesetnosy: + nascheme
messages: + msg388002
2021-03-03 06:05:28brandtbuchersetkeywords: + patch
stage: patch review
pull_requests: + pull_request23493
2021-03-03 05:48:48brandtbuchersetmessages: + msg388001
2021-03-03 05:35:10brandtbuchersetnosy: + brandtbucher
messages: + msg388000
2021-03-03 04:29:04gregory.p.smithcreate