classification
Title: OpenSSL 1.1.0c test failures
Type: behavior Stage: resolved
Components: SSL, Tests Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: christian.heimes Nosy List: Chi Hsuan Yen, Dima.Tisnek, christian.heimes, cstratak, doko, ned.deily
Priority: Keywords:

Created on 2016-11-14 14:10 by christian.heimes, last changed 2017-01-26 13:47 by Dima.Tisnek. This issue is now closed.

Messages (13)
msg280776 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-14 14:10
OpenSSL 1.1.0c broke a bunch of tests. The same tests are passing fine with OpenSSL 1.1.0 to 1.1.0b. It looks like a problem with EOF / connection close error. I'm seeing similar problems in MIT KRB5's OpenSSL plugin, too.

======================================================================
ERROR: test_ciphers (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1658, in test_ciphers
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1483, in test_connect
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_cadata (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1600, in test_connect_cadata
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_capath (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1579, in test_connect_capath
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_connect_with_context (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1541, in test_connect_with_context
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1645, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1830, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1215, in get_server_certificate
    with  create_connection(addr) as sock:
  File "/home/heimes/dev/python/cpython/Lib/socket.py", line 722, in create_connection
    raise err
  File "/home/heimes/dev/python/cpython/Lib/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_session_handling (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3552, in test_session_handling
    s.connect((HOST, server.port))
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_tls_unique_channel_binding (test.test_ssl.ThreadedTests)
Test tls-unique channel binding.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3167, in test_tls_unique_channel_binding
    s.connect((HOST, server.port))
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused
msg280778 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-14 14:19
test_server_accept (test.test_ssl.ThreadedTests) ... Exception in thread Thread-348:
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/heimes/dev/python/cpython/Lib/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3044, in serve
    remote.recv(1)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 987, in recv
    return self.read(buflen)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 865, in read
    return self._sslobj.read(len, buffer)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 627, in read
    v = self._sslobj.read(len)
OSError: [Errno 0] Error
msg280785 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-14 16:02
A git bisect between OpenSSL_1_1_0b (good) and OpenSSL_1_1_0c (bad) revealed the breaking commit:

$ git bisect good
122580ef71e4e5f355a1a104c9bfb36feee43759 is the first bad commit
commit 122580ef71e4e5f355a1a104c9bfb36feee43759
Author: Matt Caswell <matt@openssl.org>
Date:   Fri Oct 21 13:25:19 2016 +0100

    A zero return from BIO_read()/BIO_write() could be retryable
    
    A zero return from BIO_read()/BIO_write() could mean that an IO operation
    is retryable. A zero return from SSL_read()/SSL_write() means that the
    connection has been closed down (either cleanly or not). Therefore we
    should not propagate a zero return value from BIO_read()/BIO_write() back
    up the stack to SSL_read()/SSL_write(). This could result in a retryable
    failure being treated as fatal.
    
    Reviewed-by: Richard Levitte <levitte@openssl.org>
    (cherry picked from commit 4880672a9b41a09a0984b55e219f02a2de7ab75e)

:040000 040000 8097bc37a0a2a3c1e6a8879ad49ee773001d8d52 8083927cb2eb28a71baa8b90b07b0962016d74b3 M      ssl
msg280786 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-14 16:04
https://github.com/openssl/openssl/commit/122580ef71e4e5f355a1a104c9bfb36feee43759
msg280787 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-14 16:13
OpenSSL upstream bug: https://github.com/openssl/openssl/issues/1919
msg281391 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2016-11-21 21:52
Fixed upstream:
https://github.com/openssl/openssl/commit/beacb0f0c1ae7b0542fe053b95307f515b578eb7
msg281727 - (view) Author: Charalampos Stratakis (cstratak) * Date: 2016-11-25 15:33
Tested this in Fedora Rawhide virtual machine, where the fix for the problematic openssl commit was backported, and now the tests hang at test_poplib.

Exception in thread Thread-982:
Traceback (most recent call last):
  File "/home/harris/dev/cpython/_install/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/harris/dev/cpython/_install/lib/python3.6/test/test_poplib.py", line 222, in run
    asyncore.loop(timeout=0.1, count=1)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 207, in loop
    poll_fun(timeout, map)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 150, in poll
    read(obj)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 87, in read
    obj.handle_error()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 423, in handle_read_event
    self.handle_read()
  File "/home/harris/dev/cpython/_install/lib/python3.6/test/test_poplib.py", line 192, in handle_read
    asynchat.async_chat.handle_read(self)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asynchat.py", line 121, in handle_read
    self.handle_error()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asynchat.py", line 117, in handle_read
    data = self.recv(self.ac_in_buffer_size)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 374, in recv
    data = self.socket.recv(buffer_size)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 987, in recv
    return self.read(buflen)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 865, in read
    return self._sslobj.read(len, buffer)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 627, in read
    v = self._sslobj.read(len)
OSError: [Errno 0] Error
msg281894 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-11-28 17:09
The test suite is passing with OpenSSL 1.1.0d-dev (OpenSSL_1_1_0-stable branch). I consider 1.1.0c a broken and unsupported release.
msg283248 - (view) Author: Matthias Klose (doko) * (Python committer) Date: 2016-12-15 07:30
Fyi, Debian's 1.1.0c is fixed, so please don't blacklist this version in general.
msg283285 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-12-15 11:41
No blacklisting :)
I don't want to mess with any code prior to the final release of Python 3.6.0. Let's just document that stock 1.1.0c is not compatible.

Ned, I'm setting the release blocker flag as a reminder to add a comment to the release notes of 3.6.0.
msg283364 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2016-12-16 04:01
I added the following text to the "Notes on this release" section for the 3.6.0rc1 release page and will copy it to future 3.6.x release pages until sometime after a fixed 1.1.0 is released:

"If you are building Python from source, beware that the OpenSSL 1.1.0c release, the most recent as of this update, is known to cause Python 3.6 test suite failures and its use should be avoided without additional patches. It is expected that the next release of the OpenSSL 1.1.0 series will fix these problems. See http://bugs.python.org/issue28689 for more information."

https://www.python.org/downloads/release/python-360rc1/

I'm closing this issue for now; feel free to reopen if there is something else you plan to do with it.
msg283368 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2016-12-16 06:48
Thanks, Ned!

To clarify, it's not jus the test suite. OpenSSL 1.1.0c broke the ssl module.
msg286312 - (view) Author: Dima Tisnek (Dima.Tisnek) Date: 2017-01-26 13:47
1.1.0d is due today. Who wants to test it? :)
History
Date User Action Args
2017-01-26 13:47:50Dima.Tisneksetnosy: + Dima.Tisnek
messages: + msg286312
2016-12-16 06:48:48christian.heimessetmessages: + msg283368
2016-12-16 04:01:28ned.deilysetstatus: open -> closed
priority: release blocker ->
messages: + msg283364

resolution: fixed
stage: resolved
2016-12-15 11:41:11christian.heimessetpriority: high -> release blocker
nosy: + ned.deily
messages: + msg283285

2016-12-15 07:30:10dokosetnosy: + doko
messages: + msg283248
2016-11-28 17:09:00christian.heimessetmessages: + msg281894
2016-11-25 15:33:56cstrataksetmessages: + msg281727
2016-11-21 21:52:19cstrataksetnosy: + cstratak
messages: + msg281391
2016-11-14 16:13:18christian.heimessetmessages: + msg280787
2016-11-14 16:04:28christian.heimessetmessages: + msg280786
2016-11-14 16:02:45christian.heimessetmessages: + msg280785
2016-11-14 14:19:35christian.heimessetmessages: + msg280778
2016-11-14 14:19:18Chi Hsuan Yensetnosy: + Chi Hsuan Yen
2016-11-14 14:10:47christian.heimescreate