classification
Title: test_asyncio fails on RHEL8, or on Fedora using NEXT security policy
Type: Stage: resolved
Components: asyncio, Tests Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, cstratak, vstinner, yselivanov
Priority: normal Keywords: patch

Created on 2018-11-29 16:13 by cstratak, last changed 2018-11-30 22:00 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
asyncio_failures_3.8.log cstratak, 2018-11-29 16:13
asyncio_failures_3.7.log cstratak, 2018-11-29 16:13
asyncio_failures_3.6.log cstratak, 2018-11-29 16:14
Pull Requests
URL Status Linked Edit
PR 10826 merged cstratak, 2018-11-30 16:21
PR 10830 closed miss-islington, 2018-11-30 16:57
PR 10831 merged vstinner, 2018-11-30 17:04
PR 10832 merged vstinner, 2018-11-30 17:24
PR 10834 merged vstinner, 2018-11-30 17:32
Messages (16)
msg330698 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-29 16:13
When compiling python3 on RHEL8 and run the tests, test_asyncio fails with 8 failures (and more on older branches).

openssl version is 1.1.1

Attaching the failures for the various branches.
msg330699 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-29 16:14
Seems that 3.8 and 3.7 get the same failures, while 3.6 gets 14 instead of 8.
msg330767 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-11-30 10:24
The strange thing is these tests are passed on our build bots.

In logs I see only ConnectionResetError. Can it be related to your configuration?
msg330787 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-30 14:28
It seems I can reproduce it on Fedora as well by setting stronger crypto defaults through 'update-crypto-policies --set FUTURE'.

Repo located here: https://gitlab.com/redhat-crypto/fedora-crypto-policies/tree/master

The changes are many, but if I compare with RHEL8, the minimal changes that could affect it are:

-# DH params size: >= 1023
+# DH params size: >= 2048

-# TLS protocols: TLS >= 1.0
+# TLS protocols: TLS >= 1.2, DTLS >= 1.2

-@protocol_list = ('TLS1.3', 'TLS1.2', 'TLS1.1', 'TLS1.0', 'DTLS1.2', 'DTLS1.0');
+@protocol_list = ('TLS1.3', 'TLS1.2', 'DTLS1.2');

- $min_tls_version = 'TLS1.0';
- min_dtls_version = 'DTLS1.0';
+ $min_tls_version = 'TLS1.2';
+ $min_dtls_version = 'DTLS1.2';

# Parameter sizes
- $min_dh_size = 1023;
+ $min_dh_size = 2048;
msg330788 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-11-30 14:44
Maybe this is the reason.
asycio test suite uses these ssl contexts: https://github.com/python/cpython/blob/master/Lib/test/test_asyncio/utils.py#L72-L92

Maybe bumping used protocol version will help to pass tests on your box.
Would you try it?
msg330790 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-30 15:13
I believe I figured out the issue, at least on the master branch.

While checking the certificates used by asyncio tests within the test_asyncio folder I noticed they were quite outdated when compared to the more recent updated ones with the test/ folder, which take into account the stronger crypto defaults introduced in the latest openssl versions.

And by looking at https://github.com/python/cpython/commit/6d8c1abb003a4cb05f1ddcf0eeddeeeed513cd57#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

The immediate workaround would be to just copy over the certs but a better approach would be to just reuse the certs within the test/ folder instead of relying on copying them over to test_asyncio/
msg330803 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 16:26
> The strange thing is these tests are passed on our build bots.

RHEL8 has a very strict security policy by default. I'm not sure if any OS run on buildbot has a security policy as strict as RHEL8?

> Maybe bumping used protocol version will help to pass tests on your box.

I tried to tune the SSLContext in many different ways but it doesn't work. The problem comes from the .pem files.

I confirm that copying .pem files from Lib/test/ into Lib/test/test_asyncio/ does fix the issue.

> And by looking at https://github.com/python/cpython/commit/6d8c1abb003a4cb05f1ddcf0eeddeeeed513cd57#diff-a8e7dbb528601706db0f01d01332bb76 it seems that those certs are just copied from test/ within test_asyncio/. So by copying over the old certs, the tests actually pass.

In this case, I don't see the point of having two copies of the same files.

PR 10826 does the right fix: remove .pem files from Lib/test/test_asyncio/ and reuse .pem files from Lib/test/.
msg330804 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2018-11-30 16:29
Also on Fedora the same set of security policies can be set as RHEL8 by utilizing 'update-crypto-policies --set NEXT'
msg330808 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 16:56
New changeset b062ba77b617b0f89b7ea25d14cc77c991462ad4 by Victor Stinner (stratakis) in branch 'master':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826)
https://github.com/python/cpython/commit/b062ba77b617b0f89b7ea25d14cc77c991462ad4
msg330812 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2018-11-30 17:26
IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.
msg330813 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 17:28
> IIRC two copies exist from very early development times when asyncio was not a part of Python stdlib.

Yeah, that was my guess as well. Maybe data_file() could be simplified or replaced by support.findfile(), but I chose the easy solution (minimize changes) :-)
msg330814 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 17:30
New changeset 7212148c95947b0fdfcb0c8e37d4357287bdb4bd by Victor Stinner in branch 'master':
bpo-35352: Cleanup test_asyncio/utils.py (GH-10831)
https://github.com/python/cpython/commit/7212148c95947b0fdfcb0c8e37d4357287bdb4bd
msg330818 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 19:44
New changeset 38bed786a219c65d5a51c7ef4ffd97e12653a095 by Victor Stinner in branch '3.7':
[3.7] bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10834)
https://github.com/python/cpython/commit/38bed786a219c65d5a51c7ef4ffd97e12653a095
msg330819 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 19:44
New changeset 02250e57c37339ea6de08ab077a307e75eef02f5 by Victor Stinner in branch '3.6':
bpo-35352: test_asyncio uses the certificate set from the test directory (GH-10826) (GH-10832)
https://github.com/python/cpython/commit/02250e57c37339ea6de08ab077a307e75eef02f5
msg330820 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 19:48
I leave the issue open until someone validates manually that the issue os fixed in all branches with stricter security (ex: RHEL8).
msg330828 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-11-30 22:00
I tested on Fedora 29 using:

   sudo update-crypto-policies --set NEXT

With this config, I was able to reproduce the test_asyncio failure on 3.6, 3.7 and master branches.

I confirm that the commits fixed test_asyncio in these 3 branches. Thanks Charalampos Stratakis!
History
Date User Action Args
2018-11-30 22:00:19vstinnersetstatus: open -> closed
title: test_asyncio fails on RHEL8 -> test_asyncio fails on RHEL8, or on Fedora using NEXT security policy
messages: + msg330828

resolution: fixed
stage: patch review -> resolved
2018-11-30 19:48:24vstinnersetmessages: + msg330820
2018-11-30 19:44:46vstinnersetmessages: + msg330819
2018-11-30 19:44:31vstinnersetmessages: + msg330818
2018-11-30 17:32:15vstinnersetpull_requests: + pull_request10073
2018-11-30 17:30:11vstinnersetmessages: + msg330814
2018-11-30 17:28:16vstinnersetmessages: + msg330813
2018-11-30 17:26:05asvetlovsetmessages: + msg330812
2018-11-30 17:24:12vstinnersetpull_requests: + pull_request10072
2018-11-30 17:04:30vstinnersetpull_requests: + pull_request10071
2018-11-30 16:57:11miss-islingtonsetpull_requests: + pull_request10070
2018-11-30 16:56:59vstinnersetmessages: + msg330808
2018-11-30 16:29:14cstrataksetmessages: + msg330804
2018-11-30 16:26:56vstinnersetnosy: + vstinner
messages: + msg330803
2018-11-30 16:21:26cstrataksetkeywords: + patch
stage: patch review
pull_requests: + pull_request10067
2018-11-30 15:13:00cstrataksetmessages: + msg330790
2018-11-30 14:44:46asvetlovsetmessages: + msg330788
2018-11-30 14:28:46cstrataksetmessages: + msg330787
2018-11-30 10:24:02asvetlovsetmessages: + msg330767
2018-11-29 16:14:40cstrataksetmessages: + msg330699
2018-11-29 16:14:08cstrataksetfiles: + asyncio_failures_3.6.log
2018-11-29 16:13:58cstrataksetfiles: + asyncio_failures_3.7.log
2018-11-29 16:13:50cstratakcreate