classification
Title: SSL tests failed due to expired svn.python.org SSL certificate
Type: Stage: resolved
Components: Tests Versions: Python 3.6, Python 3.2, Python 3.3, Python 3.4, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: martin.panter Nosy List: EWDurbin, alex, benjamin.peterson, geertj, georg.brandl, jmadden, koobs, larry, lemburg, martin.panter, python-dev, skrah, vincent-legoll, yan12125
Priority: normal Keywords: patch

Created on 2015-12-24 20:47 by yan12125, last changed 2016-03-27 04:22 by martin.panter. This issue is now closed.

Files
File name Uploaded Description Edit
svn-cert.patch martin.panter, 2016-01-04 07:43 [obsolete] review
local-server.patch martin.panter, 2016-01-06 02:18 review
local-server.v2.patch martin.panter, 2016-01-06 07:21 Change tests to use local server review
pythontest.patch martin.panter, 2016-01-08 10:42 Use self-signed.pythontest.net review
pythontest-set-ca.patch martin.panter, 2016-01-12 08:36 For pythontestdotnet repo
use-pythontest.v2.patch martin.panter, 2016-01-12 08:44 Use self-signed.pythontest.net review
local-server.v3.patch martin.panter, 2016-01-16 04:27 review
Messages (29)
msg256969 - (view) Author: Chih-Hsuan Yen (yan12125) * Date: 2015-12-24 20:47
The certificate of svn.python.org expires at Thu 24 Dec 2015 08:28:32 PM CST GMT, about 20 minutes ago. Please update its certificate or lots of tests in Lib\test\test_ssl.py fails with SSL: CERTIFICATE_VERIFY_FAILED.
msg256994 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2015-12-25 15:41
I'm here from (duplicate) issue 25950
msg256996 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2015-12-25 15:56
Regardless of the cert having expired, I think SSL tests either should use the main python.org site, e.g. https://www.python.org/ or the domain we specifically have for stdlib tests: http://pythontest.net/.

Benjamin: If you need a cert for pythontest.net, I can issue one next week.

Since the tests use CA Cert as root CA for the domain, I'm not sure whether getting a non-CA Cert for svn.python.org would be a good thing. If possible, I can issue one as well, but would have to know how svn.python.org is managed.
msg256997 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2015-12-25 15:59
I agree the tests shouldn't rely on a legacy domain like svn.python.org

In the mean time Ernest is working on getting a valid cert set up.
msg256999 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2015-12-25 16:14
There's good news and bad news, which do you want first?

Good news, great!

svn.python.org now has a certificate that's not expired, and it's even trusted by major trust stores.

Bad news?

The tests rely on the cert for svn.python.org specifically be a cacert one. IMO this is a bug in the tests and should be fixed.
msg257000 - (view) Author: Ernest W. Durbin III (EWDurbin) * (Python triager) Date: 2015-12-25 16:20
Currently svn.python.org is served from one of the last remaining "unmanaged" boxes in our infrastructure, dinsdale.python.org.

If a cacert signed certificate can be supplied, it can be installed. I see a tree of them in /etc/apache2/ssl/cacert/{2010,2011,2013}. Previously it seems that they were supplied by instantssl as i see certs from 2005 and 2007 there.

Tying ourselves to a single CA for these tests indefinitely doesn't seem to be the greatest idea though.
msg257152 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-12-29 01:16
New changeset 9d0c15425b59 by Stefan Krah in branch 'default':
Issue #25940: Make buildbots usable again until a solution is found.
https://hg.python.org/cpython/rev/9d0c15425b59
msg257181 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2015-12-29 13:00
Setting to blocker because I've disabled the offending tests.
msg257442 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-04 05:52
Assuming it is practical, would it be appropriate to convert the failing tests to connect to a local server running in a background Python thread? Perhaps using the existing ThreadedEchoServer. That way the tests wouldn’t require an internet connection, and the certificates used etc should be all contained within the Python source tree.
msg257443 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-04 07:43
In the meantime, this patch changes the tests to use the new DST root certificate. This seems to be the minimum to make the tests pass again, but someone else should probably review it because I’m not too experienced with SSL certificate stuff.
msg257452 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-01-04 10:32
On 04.01.2016 08:43, Martin Panter wrote:
> 
> In the meantime, this patch changes the tests to use the new DST root certificate. This seems to be the minimum to make the tests pass again, but someone else should probably review it because I’m not too experienced with SSL certificate stuff.

Thanks. I still think that we should use a more stable
resource for this other than svn.python.org, e.g. the
https://www.pythontest.net/ setup which was created specifically
for stdlib tests and which can more easily be adapted to
fit the needs of the tests than a public resource which is meant
for other purposes.

The local server idea may also work, but would have to run on
a non-privileged port.
msg257453 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-01-04 10:43
Can't / Shouldn't these be mocked?
msg257578 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-06 02:18
Koobs: What thing are you suggesting to mock? The remote server perhaps, which is what I want to do too?

Here is a patch that implements my local SSL server idea by reusing ThreadedEchoServer.

There is one problem with local-server.patch, and I don’t know how to fix it properly. At the end of test_bio_handshake() and test_bio_read_write_data(), the unwrap() call keeps raising SSL_ERROR_SYSCALL and the test hangs in an infinite loop. I could work around this by making the server call unwrap() before shutting the socket down. But it would be better to fix ssl_io_loop() if anyone understands what is going wrong.
msg257588 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-06 07:21
After looking into the unwrap() problem more, I think I understand the problem, and believe making the server call unwrap() is the correct way forward. SSL_ERROR_SYSCALL is Open SSL’s way of saying the connection was shut down insecurely (among other things). So I propose local-server.v2.patch which passes all the tests without hanging.

I am inclined to commit my first patch soon, which is fairly simple and should make the buildbots happy. Then people can have a chance to review the second (now third) patch, which is more involved.

Does the problem really need to be fixed in the 3.2 branch, or is 3.5+ and 2.7 good enough?
msg257589 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-01-06 08:10
As many branches as we can muster, ideally all, at least 3.4, 3.5 default and 2.7 please. Back porting is a massive pain, and downstream OS's want to retain integrity of tests to ensure quality of Python in their respective ecosystems
msg257749 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-08 10:42
In python-dev I offered to make a patch to switch to https://self-signed.pythontest.net. Here is my patch so far.

There is one outstanding problem though: test_get_ca_certs_capath() fails. I think it is because the self-signed certificate does not have a “CA basic constraints” flag set, and SSLContext.get_ca_certs() no longer lists the certificate. Maybe someone else can help. I suspect this would need to be fixed on the server side (or find another server with a certificate signed with this flag set?), but I could easily be wrong :).
msg257762 - (view) Author: Vincent Legoll (vincent-legoll) * Date: 2016-01-08 15:12
Maybe if the server change is not approved you can still push the part of the patch that un-hardwire that server name everywhere...
msg258076 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-12 08:36
This is a patch against the pythontestdotnet repository to change the certificate to have the CA flag set. It should be applied in conjunction with an (upcoming) update to the test suite, otherwise test_httplib will fail with certificate verification errors.

If I push this patch to the pythontest repository, will the server immediately update itself, or are other steps required?
msg258078 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-12 08:44
Here is the corresponding patch for the test suite. This version should fix all the SSL tests as long as the server is updated.
msg258086 - (view) Author: Marc-Andre Lemburg (lemburg) * (Python committer) Date: 2016-01-12 09:20
The patch looks good. I only have one question:

Why are you removing this part ? ...

@@ -1684,13 +1688,8 @@
             try:
                 ret = func(*args)
             except ssl.SSLError as e:
-                # Note that we get a spurious -1/SSL_ERROR_SYSCALL for
-                # non-blocking IO. The SSL_shutdown manpage hints at this.
-                # It *should* be safe to just ignore SYS_ERROR_SYSCALL because
-                # with a Memory BIO there's no syscalls (for IO at least).
                 if e.errno not in (ssl.SSL_ERROR_WANT_READ,
-                                   ssl.SSL_ERROR_WANT_WRITE,
-                                   ssl.SSL_ERROR_SYSCALL):
+                                   ssl.SSL_ERROR_WANT_WRITE):
                     raise
                 errno = e.errno
             # Get any data from the outgoing BIO irrespective of any error, and
msg258100 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-12 11:45
Marc-Andre: This is a fix or workaround for the problem I first described in <https://bugs.python.org/issue25940#msg257578>. It looks like the code was written by Geert Jansen in Issue 21965. I suspect it is not right, but I am not familiar enough with the Open SSL API to be certain.

Geert: can you shed any light on why ssl_io_loop() in /Lib/test/test_ssl.py catches SSL_ERROR_SYSCALL and immediately retries the call? I found that when the call is unwrap(), and the remote end has shut down the TCP connection without a secure SSL-level shutdown, this calls unwrap() over and over in an infinite loop.

Geert’s comment, that my latest patch removes, mentions a spurious SSL_ERROR_SYSCALL for non-blocking IO, especially for SSL_shutdown(), which is what the Python-level unwrap() method calls. Even though the OS-level socket is blocking, I guess from Open SSL’s point of view it is doing non-blocking IO through the BIO interface.

The manual page <https://www.openssl.org/docs/manmaster/ssl/SSL_shutdown.html> does mention a quirk with SSL_ERROR_SYSCALL, but only for a return value of zero, indicating the operation is half done. Python’s SSL module does not seem to pay any attention to any error codes if SSL_shutdown() returns zero; see <https://hg.python.org/cpython/annotate/v3.5.1/Modules/_ssl.c#l2051>. It just retries once, and then either returns the socket or None, never raising an exception. When the infinite loop problem occurs, the SSL_shutdown() return value is negative, indicating failure.
msg258200 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-14 13:39
New changeset 81b3beea7e99 by Georg Brandl in branch '3.2':
Issue #25940: Backport self-signed.pythontest.net testing for test_httplib
https://hg.python.org/cpython/rev/81b3beea7e99

New changeset adf750b1252d by Martin Panter in branch '3.2':
Issue #25940: Use self-signed.pythontest.net in SSL tests
https://hg.python.org/cpython/rev/adf750b1252d

New changeset e167a9296947 by Martin Panter <vadmium> in branch '3.3':
Issue #25940: Merge self-signed.pythontest.net testing from 3.2 into 3.3
https://hg.python.org/cpython/rev/e167a9296947

New changeset 32bcb9aa286e by Martin Panter <vadmium> in branch '3.4':
Issue #25940: Merge self-signed.pythontest.net testing from 3.3 into 3.4
https://hg.python.org/cpython/rev/32bcb9aa286e

New changeset 0585c0089466 by Martin Panter <vadmium> in branch '3.4':
Issue #25940: Update new SSL tests for self-signed.pythontest.net
https://hg.python.org/cpython/rev/0585c0089466

New changeset c3aa4b48b905 by Martin Panter <vadmium> in branch '3.5':
Issue #25940: Merge self-signed.pythontest.net testing from 3.4 into 3.5
https://hg.python.org/cpython/rev/c3aa4b48b905

New changeset f5f14d65297e by Martin Panter <vadmium> in branch '3.5':
Issue #25940: Update new SSL tests for self-signed.pythontest.net
https://hg.python.org/cpython/rev/f5f14d65297e

New changeset 2f2b42a8b34d by Martin Panter <vadmium> in branch 'default':
Issue #25940: Merge self-signed.pythontest.net testing from 3.5
https://hg.python.org/cpython/rev/2f2b42a8b34d

New changeset ac94418299bd by Martin Panter <vadmium> in branch 'default':
Issue #25940: test_ssl is working again
https://hg.python.org/cpython/rev/ac94418299bd
msg258203 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2016-01-14 16:30
Wow. *thank you* Georg and Martin :)
msg258253 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-15 00:05
In the 3.3 branch, I got a failure in test_ssl.ThreadedTests.test_dh_params(): “SSLError: [SSL] dh key too small (_ssl.c:548)”. But the failure also happens in the revision before my merge, so I think it must be a separate problem.

Also, I am seeing Windows 7+ buildbots are failing the new version of test_connect_ex_error(). On Linux, connecting to self-signed.pythontest.net:444 gives EHOSTUNREACH, but it seems we get ETIMEDOUT on Windows:

AssertionError: 10060 not found in (10061, 10065, 10035)

I will update the test to check for that error code as well, since all it really needs to test is that _some_ sensible error code is returned.
msg258259 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-15 01:09
New changeset fb7131939508 by Martin Panter in branch '2.7':
Issue #25940: Use self-signed.pythontest.net in SSL tests
https://hg.python.org/cpython/rev/fb7131939508
msg258262 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-01-15 02:46
New changeset b2036b717028 by Martin Panter in branch '3.2':
Issue #25940: On Windows, connecting to port 444 returns ETIMEDOUT
https://hg.python.org/cpython/rev/b2036b717028

New changeset c5cae7366835 by Martin Panter in branch '3.3':
Issue #25940: Merge ETIMEDOUT fix from 3.2 into 3.3
https://hg.python.org/cpython/rev/c5cae7366835

New changeset a806ed21bd82 by Martin Panter in branch '3.4':
Issue #25940: Merge ETIMEDOUT fix from 3.3 into 3.4
https://hg.python.org/cpython/rev/a806ed21bd82

New changeset bd9d5bc8bc95 by Martin Panter in branch '3.5':
Issue #25940: Merge ETIMEDOUT fix from 3.4 into 3.5
https://hg.python.org/cpython/rev/bd9d5bc8bc95

New changeset 32a4e7b337c9 by Martin Panter in branch 'default':
Issue #25940: Merge ETIMEDOUT fix from 3.5
https://hg.python.org/cpython/rev/32a4e7b337c9
msg258352 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-16 04:27
Here is an updated version of my local server patch. I merged it with the committed changes, and fixed a couple of mistakes. This patch does not need to be applied to the older branches.
msg262504 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-03-27 02:17
New changeset 3da812602881 by Martin Panter in branch 'default':
Issue #25940: Use internal local server more in test_ssl
https://hg.python.org/cpython/rev/3da812602881
msg262507 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-03-27 04:22
Okay so for the record, the maintainence branches were changed over from svn.python.org to self-signed.pythontest.net, and just now I changed the 3.6 branch to use a local server for most tests.
History
Date User Action Args
2016-05-13 02:46:12martin.panterlinkissue27003 superseder
2016-03-27 04:22:18martin.pantersetstatus: open -> closed
resolution: fixed
messages: + msg262507

stage: patch review -> resolved
2016-03-27 02:17:51python-devsetmessages: + msg262504
2016-01-16 04:27:32martin.pantersetpriority: release blocker -> normal
files: + local-server.v3.patch
messages: + msg258352
2016-01-15 02:46:50python-devsetmessages: + msg258262
2016-01-15 01:09:43python-devsetmessages: + msg258259
2016-01-15 00:05:38martin.pantersetmessages: + msg258253
2016-01-14 16:30:15koobssetmessages: + msg258203
2016-01-14 13:39:18python-devsetmessages: + msg258200
2016-01-12 11:45:29martin.pantersetnosy: + geertj
messages: + msg258100
2016-01-12 09:20:37lemburgsetmessages: + msg258086
2016-01-12 08:44:03martin.pantersetfiles: + use-pythontest.v2.patch

messages: + msg258078
2016-01-12 08:36:52martin.pantersetfiles: + pythontest-set-ca.patch

messages: + msg258076
2016-01-08 15:12:46vincent-legollsetmessages: + msg257762
2016-01-08 10:42:44martin.pantersetfiles: + pythontest.patch

messages: + msg257749
2016-01-07 05:35:20orsenthilsetassignee: martin.panter
2016-01-06 08:10:16koobssetmessages: + msg257589
2016-01-06 07:21:24martin.pantersetfiles: + local-server.v2.patch

messages: + msg257588
2016-01-06 02:18:24martin.pantersetfiles: + local-server.patch

messages: + msg257578
2016-01-05 14:52:01jmaddensetnosy: + jmadden
2016-01-04 20:26:38vincent-legollsetnosy: + vincent-legoll
2016-01-04 10:43:39koobssetmessages: + msg257453
2016-01-04 10:32:06lemburgsetmessages: + msg257452
2016-01-04 07:43:07martin.pantersetfiles: + svn-cert.patch
keywords: + patch
messages: + msg257443

stage: patch review
2016-01-04 05:52:02martin.pantersetnosy: + martin.panter
messages: + msg257442
2015-12-29 13:00:06skrahsetpriority: high -> release blocker
nosy: + larry, skrah, georg.brandl
messages: + msg257181

2015-12-29 01:16:08python-devsetnosy: + python-dev
messages: + msg257152
2015-12-25 16:20:45EWDurbinsetnosy: + EWDurbin
messages: + msg257000
2015-12-25 16:14:49alexsetmessages: + msg256999
2015-12-25 15:59:21alexsetnosy: + alex
messages: + msg256997
2015-12-25 15:56:33lemburgsetnosy: + lemburg, benjamin.peterson
messages: + msg256996
2015-12-25 15:41:36koobssetpriority: normal -> high
nosy: + koobs
messages: + msg256994

2015-12-25 15:24:31berker.peksaglinkissue25950 superseder
2015-12-24 20:47:46yan12125create