https://github.com/python/cpython/pull/20442/checks?check_run_id=2018900756

 ssl.SSLError: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:1122)
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1122)
 ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:1122)
 ssl.SSLError: [SSL] internal error (_ssl.c:1122)
 ssl.SSLError: [SSL] called a function you should not call (_ssl.c:1122)
 ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:1122)
 ssl.SSLError: [SSL: UNEXPECTED_MESSAGE] unexpected message (_ssl.c:1122)
ssl.SSLError: [SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:1122)
[SSL: NO_PROTOCOLS_AVAILABLE] no protocols available (_ssl.c:1122)

... and so on ...

This CI failure is preventing any PR from being merged.

Do we have a bad test certificate in the tree we need to update?
Or did some Ubuntu CI infrastructure just fall over and start rejecting a certificate it used to accept?

I believe this started around ~20210302T1500 UTC.

It seems that GitHub recently changed their "ubuntu-latest" image from Ubuntu 18.04 to Ubuntu 20.04.

A good temporary workaround would probably be to change this line:

https://github.com/python/cpython/blob/727a68b6e592eada5a65935de5c8428ef50e8741/.github/workflows/build.yml#L130

...to read "ubuntu-18.04".

I forgot to mention that I confirmed that the last passing test run used 18.04 (click "set up job" -> "Operating System" to see):

https://github.com/python/cpython/runs/2013210763?check_suite_focus=true

The next one, which started the current chain of failures, used 20.04:

https://github.com/python/cpython/runs/2018900756?check_suite_focus=true

I'm still not sure *why* this broke CI, though. Perhaps our cached OpenSSL is no longer valid?

I think it may be related to bpo-41561.  There is a bug in the Ubuntu tracker as well:

https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1899878

I agree with the temporary fix to use "ubuntu-18.04" for CI testing.

Thanks for the quick workaround!

The problem could be caused by a downstream patch in Ubuntu's OpenSSL version. Vanilla OpenSSL doesn't fail like that.

Ubuntu 20.04+ compile OpenSSL with default security level set to 2, and further customized security level 2 to prohibit TLS below v1.2 and DTLS below v1.2.

You can export custom openssl configuration that sets security level back to 1, which is compatible across any openssl series.

```
export OPENSSL_CONF=`pwd`/openssl.cnf
cat openssl.cnf
openssl_conf = default_conf
[default_conf]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
CipherString = DEFAULT@SECLEVEL=1
```

Or you can use native APIs to reset the security level to 1 in the test-suite. I.e. via the SSL_CTX_set_security_level api binding.

This is documented behaviour in Ubuntu manpages of OpenSSL and on Ubuntu Discourse

https://manpages.ubuntu.com/manpages/focal/en/man3/SSL_CTX_set_security_level.3ssl.html

https://discourse.ubuntu.com/t/default-to-tls-v1-2-in-all-tls-libraries-in-20-04-lts/12464/8

OpenSSL upstream for 3.0.0 series are refusing to bump minimum required protocol versions to prohibit out of the box old version of TLS and also don't have a standard way to disable this. Hence implementation is different in Debian, Ubuntu and Fedora. Debian's implementation is buggy with respect to DTLS and default openssl.cnf breaks 1.0.2x series libssl. And as far as I know Fedora implementation requires use of crypto-policies package which is quite advanced and not trivial to integrate in smaller environments.

BTW.

It would be advisable for Python3 to start enforcing security level 2, and prohibit DTLS v1.1 and lower by default too. By configuring openssl library on the host with setting security level, and/or setting min versions (if openssl on the host supports such api). Because allowing to use TLS v1.1 and lower out of the box is irresponsible.

Dimitri, thanks for your feedback.

I'm very well aware of the crypto policy settings and security level settings. The problem is not the fact that Ubuntu sets a higher security level and disables insecure TLS versions. The problem is the way how Ubuntu has implemented the policy to enforce the crypto settings.

Other Linux distributions like Debian and Fedora also raise the security level and disable TLS 1.0 and 1.1. Python's test suite introspects OpenSSL settings and skips tests accordingly. test_ssl is passing fine on Debian testing (updated 15 minutes ago) and Fedora 33 with similar crypto policies. Since the tests are working fine on Debian, Fedora, RHEL/CentOS, vanilla OpenSSL, our OpenSSL builds on macOS and Windows, and other Linux distros, the issue is likely caused by a downstream discrepancy in Ubuntu.

# Python main branch on Fedora 33
$ ./python 
Python 3.10.0a5+ (heads/master:cd80f430daa, Feb 24 2021, 19:44:57) 
[GCC 10.2.1 20201125 (Red Hat 10.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version
<TLSVersion.TLSv1_2: 771>
>>> ctx.security_level
2

# Python main branch on Debian testing
$ ./python 
Python 3.10.0a6+ (heads/master:94894dd45e, Mar  3 2021, 09:11:22) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> ctx = ssl.create_default_context()
>>> ctx.minimum_version
<TLSVersion.TLSv1_2: 771>
>>> ctx.security_level
2

$ ./python -m test test_ssl
0:00:00 load avg: 0.89 Run tests sequentially
0:00:00 load avg: 0.89 [1/1] test_ssl

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2.6 sec
Tests result: SUCCESS

# dpkg -l openssl
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-====================================================
ii  openssl        1.1.1j-1     amd64        Secure Sockets Layer toolkit - cryptographic utility

> It would be advisable for Python3 to start enforcing security level 2, and prohibit DTLS v1.1 and lower by default too. By configuring openssl library on the host with setting security level, and/or setting min versions (if openssl on the host supports such api). Because allowing to use TLS v1.1 and lower out of the box is irresponsible.

We are going to change the default settings in our own OpenSSL builds together with https://www.python.org/dev/peps/pep-0644/ . For Linux distros we will rely on distro-wide crypto policies.

I have backported the workaround to 3.7, 3.8, and 3.9. There was some issue with the backport bot and I didn't have time to investigate. PRs are:

https://github.com/python/cpython/pull/24716
https://github.com/python/cpython/pull/24717
https://github.com/python/cpython/pull/24718

Downstream has asked me to file a separate bug for internal error during handshake. The problem is tracked at https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1917625 .

@ned.deily, I think the 3.7 backport needs RM approval (or something?):

https://github.com/python/cpython/pull/24716

The others branches are fine now... thanks, Christian!

3.7 backport is now merged, too.

How do I get a CI run on a PR to actually pickup this change?  clicking rerun failed.

You may have to refresh the PR so that is uses the latest HEAD that includes the config change.

Yeah, that's the annoying part. Users have to rebase all their PRs in order to make CI pass. It's going to be painful. :(

Closing and reopening may work, or pushing an empty commit. I know that's helped appease some GitHub CI weirdness in the past.

yeah i figured it might require a rebase.  if anyone has the appropriate git command to do that to a branch, creating a 2-4 step CLI playbook for people to apply to pending PR branches would be useful.

we may not need that: Closing and reopening the PR worked.  I suspect the github CI "rerun" button was "rerun in the exact same config".

It seems it is enough to make a new commit.  The CI seems to re-base and re-run the PR.  At least, it worked on two of my PRs.

CI is passing again.