classification
Title: test_ssl fails with OpenSSL 1.1.1e
Type: Stage: resolved
Components: SSL Versions: Python 3.9, Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: benjamin.peterson, christian.heimes, cstratak, jmadden, koobs, mscastanho, vstinner
Priority: normal Keywords: patch

Created on 2020-03-19 20:55 by benjamin.peterson, last changed 2020-03-30 05:26 by koobs. This issue is now closed.

Files
File name Uploaded Description Edit
amd64.freebsd.shared.3.x.build.450.test.stdio.txt koobs, 2020-03-23 15:18
Pull Requests
URL Status Linked Edit
PR 19129 closed cstratak, 2020-03-24 00:36
Messages (13)
msg364638 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-03-19 20:55
======================================================================
ERROR: test_ciphers (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2120, in test_ciphers
    s.connect(self.server_addr)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 1944, in test_connect
    s.connect(self.server_addr)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_cadata (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2062, in test_connect_cadata
    s.connect(self.server_addr)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_capath (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2041, in test_connect_capath
    s.connect(self.server_addr)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_with_context (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2002, in test_connect_with_context
    s.connect(self.server_addr)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2107, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 2272, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1484, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_session_handling (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 4346, in test_session_handling
    s.connect((HOST, server.port))
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_tls_unique_channel_binding (test.test_ssl.ThreadedTests)
Test tls-unique channel binding.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/benjamin/repos/cpython/Lib/test/test_ssl.py", line 3922, in test_tls_unique_channel_binding
    s.connect((HOST, server.port))
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/benjamin/repos/cpython/Lib/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

----------------------------------------------------------------------
msg364639 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2020-03-19 23:19
I think this is caused by their EOF change: https://github.com/openssl/openssl/issues/10880 read() now sometimes gives you the infamous OSError errno 0 rather than b'' at the end of the stream.
msg364821 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-03-22 20:24
The relevant info which seems to make the tests fail:

Properly detect EOF while reading in libssl. Previously if we hit an EOF while reading in libssl then we would report an error back to the application (SSL_ERROR_SYSCALL) but errno would be 0. We now add an error to the stack (which means we instead return SSL_ERROR_SSL) and therefore give a hint as to what went wrong.

Upstream PR: https://github.com/openssl/openssl/pull/10882

urllib3 issue: https://github.com/urllib3/urllib3/issues/1825
msg364846 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-23 12:25
bpo-40044 was marked as a duplicate of this issue. Copy of Charalampos's message:
"""
The fedora rawhide buildbots started failing due to the latest update of openssl to version 1.1.1e.

e.g. https://buildbot.python.org/all/#/builders/607/builds/137

Changelog: https://www.openssl.org/news/cl111.txt

The relevant info which seems to make the tests fail:

Properly detect EOF while reading in libssl. Previously if we hit an EOF while reading in libssl then we would report an error back to the application (SSL_ERROR_SYSCALL) but errno would be 0. We now add an error to the stack (which means we instead return SSL_ERROR_SSL) and therefore give a hint as to what went wrong.

Upstream PR: https://github.com/openssl/openssl/pull/10882

urllib3 issue: https://github.com/urllib3/urllib3/issues/1825
"""
msg364852 - (view) Author: Matheus Castanho (mscastanho) Date: 2020-03-23 14:48
Hi, I believe this is the exact same problem as reported by https://bugs.python.org/issue39787

Both issues could be closed by the same fix =)
msg364854 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-03-23 14:52
I closed bpo-39787 "test_ssl and test_urllib2_localnet failing with new OpenSSL" as a duplicate of this issue. Copy of the message:

"""
test_ssl and test_urllib2_localnet are failing when Python is built against top-of-tree OpenSSL. I'm attaching the output of: `regrtest.py test_ssl test_urllib2_localnet -W`

The output is from a powerpc64le machine with Python 3.8.2+ (1bbb81b251bc) and OpenSSL master (db943f43a60d1b).

A git bisect showed the problems started with the following OpenSSL commit:

commit db943f43a60d1b5b1277e4b5317e8f288e7a0a3a
Author: Matt Caswell <matt@openssl.org>
Date:   Fri Jan 17 17:39:19 2020 +0000

    Detect EOF while reading in libssl
    
    If we hit an EOF while reading in libssl then we will report an error
    back to the application (SSL_ERROR_SYSCALL) but errno will be 0. We add
    an error to the stack (which means we instead return SSL_ERROR_SSL) and
    therefore give a hint as to what went wrong.
    
    Contains a partial fix for #10880
    
    Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
    Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
    (Merged from https://github.com/openssl/openssl/pull/10882)

This also looks similar to: https://bugs.python.org/issue28689
"""

There is a file attached which contains test failures like these ones:

test_ciphers (test.test_ssl.SimpleBackgroundTests) ...  server:  new connection from ('127.0.0.1', 39550)
 server: connection cipher is now ('TLS_AES_256_GCM_SHA384', 'TLSv1.3', 256)
 server: selected protocol is now None
Test server failure:
Traceback (most recent call last):
   File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/test/test_ssl.py", line 2388, in run
    msg = self.read()
   File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/test/test_ssl.py", line 2365, in read
    return self.sslconn.read()
   File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1101, in read
    return self._sslobj.read(len)
 OSError: [Errno 0] Error
ERROR

======================================================================
ERROR: test_connect_capath (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/test/test_ssl.py", line 2038, in test_connect_capath
    s.connect(self.server_addr)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1342, in connect
    self._real_connect(addr, False)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1333, in _real_connect
    self.do_handshake()
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer



======================================================================
ERROR: test_https (test.test_urllib2_localnet.TestUrlopen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/test/test_urllib2_localnet.py", line 560, in test_https
    data = self.urlopen("https://localhost:%s/bizarre" % handler.port, context=context)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/test/test_urllib2_localnet.py", line 464, in urlopen
    l.extend(f.readlines(200))
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/http/client.py", line 655, in readline
    result = self.fp.readline(limit)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/home/mscastanho/AT/next/opt/at-next-14.0-0-alpha/lib64/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
ssl.SSLError: [SSL: KRB5_S_TKT_NYV] unexpected eof while reading (_ssl.c:2607)
msg364856 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2020-03-23 15:18
Confirming on FreeBSD CURRENT buildbot worker which had its openssl updated to 1.1.1e a number of days ago. I reverted back to 1.1.1d

Full log of the failing build/tests with 1.1.1e attached
msg364912 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-03-24 00:38
Still searching the issue and created a first draft PR. With it, tesT_ssl and test_imaplib pass now, urllib2_localnet still has issues.
msg364994 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-03-25 15:55
This behavior change is considered being reverted upstream.

PR: https://github.com/openssl/openssl/pull/11400
msg365148 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-03-27 12:51
The change has been reverted upstream.

Also on the rawhide buildbots, we have an updated build with the commit reverted, so they returned back to green.

Now the revertion will be included at a new release of the 1.1.1 branch, however it will still be on the 3.0.0 branch when it will be released, so at that point a possible rewrite of the ssl module will be required, as is the case with every major openssl update.

I'd suggest to close this issue and open a new one for compatibility with openssl 3.0.0.
msg365179 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2020-03-27 18:14
And there is already a meta-issue created by cheimes for 3.0.0: https://bugs.python.org/issue38820
msg365294 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2020-03-30 03:52
Updating the koobs-freebsd-564d buildbots FreBSD openssl port now which has had the 1.1.1e EOF change reverted. Will run the last 3.x build again and report here whether it passes (it should) or not.
msg365297 - (view) Author: Kubilay Kocak (koobs) (Python triager) Date: 2020-03-30 05:26
Confirming: test_ssl passes with 1.1.1e + reverted EOF change
History
Date User Action Args
2020-03-30 05:26:27koobssetmessages: + msg365297
2020-03-30 03:52:48koobssetmessages: + msg365294
2020-03-29 02:59:42benjamin.petersonsetstatus: open -> closed
resolution: wont fix
stage: patch review -> resolved
2020-03-27 18:14:42cstrataksetmessages: + msg365179
2020-03-27 12:51:21cstrataksetmessages: + msg365148
2020-03-25 15:55:14cstrataksetmessages: + msg364994
2020-03-25 14:41:28jmaddensetnosy: + jmadden
2020-03-24 00:38:45cstrataksetmessages: + msg364912
2020-03-24 00:36:32cstrataksetkeywords: + patch
stage: patch review
pull_requests: + pull_request18491
2020-03-23 15:18:44koobssetfiles: + amd64.freebsd.shared.3.x.build.450.test.stdio.txt
nosy: + koobs
messages: + msg364856

2020-03-23 14:52:54vstinnersetmessages: + msg364854
2020-03-23 14:52:51vstinnerlinkissue39787 superseder
2020-03-23 14:48:32mscastanhosetnosy: + mscastanho
messages: + msg364852
2020-03-23 12:25:13vstinnersetnosy: + vstinner
messages: + msg364846
2020-03-22 20:24:51cstrataksetnosy: + cstratak
messages: + msg364821
2020-03-19 23:19:18benjamin.petersonsetmessages: + msg364639
2020-03-19 20:55:46benjamin.petersoncreate