classification
Title: Workaround or fix for SSL ".._KEY_TOO_SMALL" test failures
Type: crash Stage: patch review
Components: Tests Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: christian.heimes, larry
Priority: release blocker Keywords: patch

Created on 2020-07-01 14:35 by larry, last changed 2020-07-16 06:00 by larry.

Files
File name Uploaded Description Edit
failures larry, 2020-07-02 07:27
test_ssl_failure larry, 2020-07-02 07:44
test_ssl_36_branch larry, 2020-07-02 07:47
openssl.revision.history.txt larry, 2020-07-02 07:52
test_ssl_verbose_36_master larry, 2020-07-02 08:04
Pull Requests
URL Status Linked Edit
PR 21258 merged christian.heimes, 2020-07-01 15:02
PR 21278 merged christian.heimes, 2020-07-02 08:12
Messages (18)
msg372755 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-01 14:35
I'm testing 3.5.10rc1 on a freshly installed Linux (Pop!_OS 20.04), and I'm getting a lot of these test failures:

ssl.SSLError: [SSL: EE_KEY_TOO_SMALL] ee key too small (_ssl.c:2951)

Apparently the 2048 keys used in the tests are considered "too small" with brand-new builds of the SSL library.

Christian: you upgraded the test suite keys to 3072 bits back in 2018 (issue #34542), but didn't backport this as far as 3.5 because it was in security-fixes-only mode.  I experimented with taking your patch to 3.6 and applying it to 3.5, but 80% of the patches didn't apply cleanly.  Could you either backport this upgrade to 3.5 (I'll happily accept the PR), or advise me on how to otherwise mitigate the problem?  I don't really want to turn off all those tests.  Thanks!
msg372756 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-01 14:44
I'll look into it.
msg372810 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:07
New changeset d565be84993a3d618add139cf21038e12c60a13e by Christian Heimes in branch '3.5':
bpo-41183: Update test certs and keys (#21258)
https://github.com/python/cpython/commit/d565be84993a3d618add139cf21038e12c60a13e
msg372812 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:09
Thanks for the backport!
msg372815 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:27
Christian:  Help!  Again!

I merged your PR, pulled a fresh copy, built it, and ran the test suite.  I get seven failures in I think the same modules.

Most of the failures are either "ssl.SSLError: [SSL] internal error (_ssl.c:728)", or some flavor of "OSError: [Errno 0] Error".  Sadly not helpful.

But!  Occasionally the test suite prints a very telling error:

ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:3233)

Attached is the output of running just those seven tests.  (One test is now working, not sure why.)

Obviously these tests pass on the buildbots, I assume that's because their OpenSSL is slightly older.  But I don't think I can ship 3.5.10rc1 if it won't build with current OpenSSL.

You should be able to simply pull the current 3.5 head (d565be84993a3d618add139cf21038e12c60a13e) to reproduce the error.
msg372816 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:28
Upgrading to release blocker.
msg372817 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-02 07:40
I'm testing with latest build of OpenSSL 1.1.1 and Fedora's DEFAULT crypto policy here. Your vendor may have configured OpenSSL with a more strict crypto policy. 

Could you please attach a full output of ./python -m test -v test_ssl?

Does the 3.6 test suite pass on your test machine?
msg372818 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:44
test_ssl was one of the seven modules that failed.  But attached here is just the output of

% ./python -m test -v test_ssl >& test_ssl_failure
msg372819 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:47
The 3.6 branch of python/cpython fails as well on this machine.  Output attached.
msg372820 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 07:52
I assume this is building against the system OpenSSL.  On this machine, the "openssl", "libssl1.1", and "libssl-dev" packages are all version "1.1.1f-1ubuntu2".

The OS is "Pop!_OS" version 20.04, which is a derivative of Ubuntu 20.04.  It appears to be getting this package straight out of the Ubuntu package repo.  The maintainer is listed as "Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>".

Attached is the revision history, copied and pasted out of the package manager's "changelog".
msg372822 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-02 08:02
test_ssl_36_branch just contains "1 test failed: test_ssl". Could you please attach a verbose run?

The problems are caused by security policy. We had similar issues in Fedora.

    - Set OPENSSL_TLS_SECURITY_LEVEL=2 as compiled-in minimum security
      level. Change meaning of SECURITY_LEVEL=2 to prohibit TLS versions
      below 1.2 and update documentation. Previous default of 1, can be set
      by calling SSL_CTX_set_security_level(), SSL_set_security_level() or
      using ':@SECLEVEL=1' CipherString value in openssl.cfg.

I can fix "SSL: DH_KEY_TOO_SMALL" in another PR. The other issues are harder to fix.
msg372824 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 08:04
./python -m test -v test_ssl >& test_ssl_verbose_36_master
msg372825 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 08:05
Do you need a temporary login on one of my Pop!_OS computers, in order to test?
msg372826 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-02 08:14
GH-21278 takes care of test failures related to DH params.

For the other test failures somebody has to backport df6ac7e2b82d921a6e9ff5571b40c6dbcf635581 to 3.6 and 3.5. I cannot promise that I'm able to find time to do the backport today.
msg372827 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 08:16
Gotcha.  Thanks for looking into it for me.  I don't think the world is super anxious about getting 3.5.10rc1 so it's not a big huge deal.  But I will wait to hear back from you.  Thanks!
msg372850 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-02 12:32
New changeset f52bf62fe12d46267e958f80dbe1f4425b55cd0f by Christian Heimes in branch '3.5':
bpo-41183: Update finite DH params to 3072 bits (#21278)
https://github.com/python/cpython/commit/f52bf62fe12d46267e958f80dbe1f4425b55cd0f
msg373396 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-09 12:17
Any news?
msg373744 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2020-07-16 06:00
Ping?
History
Date User Action Args
2020-07-16 06:00:12larrysetmessages: + msg373744
2020-07-09 12:17:37larrysetmessages: + msg373396
2020-07-02 12:32:22larrysetmessages: + msg372850
2020-07-02 08:16:20larrysetmessages: + msg372827
2020-07-02 08:14:13christian.heimessetmessages: + msg372826
2020-07-02 08:12:28christian.heimessetstage: needs patch -> patch review
pull_requests: + pull_request20427
2020-07-02 08:05:41larrysetmessages: + msg372825
2020-07-02 08:04:07larrysetfiles: + test_ssl_verbose_36_master

messages: + msg372824
2020-07-02 08:02:25christian.heimessetmessages: + msg372822
2020-07-02 07:52:58larrysetfiles: + openssl.revision.history.txt

messages: + msg372820
2020-07-02 07:47:50larrysetfiles: + test_ssl_36_branch

messages: + msg372819
2020-07-02 07:44:53larrysetfiles: + test_ssl_failure

messages: + msg372818
2020-07-02 07:40:26christian.heimessetmessages: + msg372817
2020-07-02 07:28:22larrysetstatus: closed -> open
priority: high -> release blocker
messages: + msg372816

resolution: fixed ->
stage: resolved -> needs patch
2020-07-02 07:27:53larrysetfiles: + failures

messages: + msg372815
title: Workaround or fix for SSL "EE_KEY_TOO_SMALL" test failures -> Workaround or fix for SSL ".._KEY_TOO_SMALL" test failures
2020-07-02 07:09:19larrysetstatus: open -> closed
resolution: fixed
messages: + msg372812

stage: patch review -> resolved
2020-07-02 07:07:19larrysetmessages: + msg372810
2020-07-01 15:02:01christian.heimessetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request20405
2020-07-01 14:44:32christian.heimessetmessages: + msg372756
2020-07-01 14:35:46larrycreate