OpenSSL 1.1.0c broke a bunch of tests. The same tests are passing fine with OpenSSL 1.1.0 to 1.1.0b. It looks like a problem with EOF / connection close error. I'm seeing similar problems in MIT KRB5's OpenSSL plugin, too.

======================================================================
ERROR: test_ciphers (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1658, in test_ciphers
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1483, in test_connect
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_cadata (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1600, in test_connect_cadata
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_connect_capath (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1579, in test_connect_capath
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_connect_with_context (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1541, in test_connect_with_context
    s.connect(self.server_addr)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1084, in _real_connect
    self.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 104] Connection reset by peer

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1645, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 1830, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1215, in get_server_certificate
    with  create_connection(addr) as sock:
  File "/home/heimes/dev/python/cpython/Lib/socket.py", line 722, in create_connection
    raise err
  File "/home/heimes/dev/python/cpython/Lib/socket.py", line 713, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_session_handling (test.test_ssl.ThreadedTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3552, in test_session_handling
    s.connect((HOST, server.port))
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused

======================================================================
ERROR: test_tls_unique_channel_binding (test.test_ssl.ThreadedTests)
Test tls-unique channel binding.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3167, in test_tls_unique_channel_binding
    s.connect((HOST, server.port))
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1093, in connect
    self._real_connect(addr, False)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 1080, in _real_connect
    socket.connect(self, addr)
ConnectionRefusedError: [Errno 111] Connection refused

test_server_accept (test.test_ssl.ThreadedTests) ... Exception in thread Thread-348:
Traceback (most recent call last):
  File "/home/heimes/dev/python/cpython/Lib/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/heimes/dev/python/cpython/Lib/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/heimes/dev/python/cpython/Lib/test/test_ssl.py", line 3044, in serve
    remote.recv(1)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 987, in recv
    return self.read(buflen)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 865, in read
    return self._sslobj.read(len, buffer)
  File "/home/heimes/dev/python/cpython/Lib/ssl.py", line 627, in read
    v = self._sslobj.read(len)
OSError: [Errno 0] Error

A git bisect between OpenSSL_1_1_0b (good) and OpenSSL_1_1_0c (bad) revealed the breaking commit:

$ git bisect good
122580ef71e4e5f355a1a104c9bfb36feee43759 is the first bad commit
commit 122580ef71e4e5f355a1a104c9bfb36feee43759
Author: Matt Caswell <matt@openssl.org>
Date:   Fri Oct 21 13:25:19 2016 +0100

    A zero return from BIO_read()/BIO_write() could be retryable
    
    A zero return from BIO_read()/BIO_write() could mean that an IO operation
    is retryable. A zero return from SSL_read()/SSL_write() means that the
    connection has been closed down (either cleanly or not). Therefore we
    should not propagate a zero return value from BIO_read()/BIO_write() back
    up the stack to SSL_read()/SSL_write(). This could result in a retryable
    failure being treated as fatal.
    
    Reviewed-by: Richard Levitte <levitte@openssl.org>
    (cherry picked from commit 4880672a9b41a09a0984b55e219f02a2de7ab75e)

:040000 040000 8097bc37a0a2a3c1e6a8879ad49ee773001d8d52 8083927cb2eb28a71baa8b90b07b0962016d74b3 M      ssl

https://github.com/openssl/openssl/commit/122580ef71e4e5f355a1a104c9bfb36feee43759

OpenSSL upstream bug: https://github.com/openssl/openssl/issues/1919

Fixed upstream:
https://github.com/openssl/openssl/commit/beacb0f0c1ae7b0542fe053b95307f515b578eb7

Tested this in Fedora Rawhide virtual machine, where the fix for the problematic openssl commit was backported, and now the tests hang at test_poplib.

Exception in thread Thread-982:
Traceback (most recent call last):
  File "/home/harris/dev/cpython/_install/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/harris/dev/cpython/_install/lib/python3.6/test/test_poplib.py", line 222, in run
    asyncore.loop(timeout=0.1, count=1)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 207, in loop
    poll_fun(timeout, map)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 150, in poll
    read(obj)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 87, in read
    obj.handle_error()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 83, in read
    obj.handle_read_event()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 423, in handle_read_event
    self.handle_read()
  File "/home/harris/dev/cpython/_install/lib/python3.6/test/test_poplib.py", line 192, in handle_read
    asynchat.async_chat.handle_read(self)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asynchat.py", line 121, in handle_read
    self.handle_error()
  File "/home/harris/dev/cpython/_install/lib/python3.6/asynchat.py", line 117, in handle_read
    data = self.recv(self.ac_in_buffer_size)
  File "/home/harris/dev/cpython/_install/lib/python3.6/asyncore.py", line 374, in recv
    data = self.socket.recv(buffer_size)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 987, in recv
    return self.read(buflen)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 865, in read
    return self._sslobj.read(len, buffer)
  File "/home/harris/dev/cpython/_install/lib/python3.6/ssl.py", line 627, in read
    v = self._sslobj.read(len)
OSError: [Errno 0] Error

The test suite is passing with OpenSSL 1.1.0d-dev (OpenSSL_1_1_0-stable branch). I consider 1.1.0c a broken and unsupported release.

Fyi, Debian's 1.1.0c is fixed, so please don't blacklist this version in general.

No blacklisting :)
I don't want to mess with any code prior to the final release of Python 3.6.0. Let's just document that stock 1.1.0c is not compatible.

Ned, I'm setting the release blocker flag as a reminder to add a comment to the release notes of 3.6.0.

I added the following text to the "Notes on this release" section for the 3.6.0rc1 release page and will copy it to future 3.6.x release pages until sometime after a fixed 1.1.0 is released:

"If you are building Python from source, beware that the OpenSSL 1.1.0c release, the most recent as of this update, is known to cause Python 3.6 test suite failures and its use should be avoided without additional patches. It is expected that the next release of the OpenSSL 1.1.0 series will fix these problems. See http://bugs.python.org/issue28689 for more information."

https://www.python.org/downloads/release/python-360rc1/

I'm closing this issue for now; feel free to reopen if there is something else you plan to do with it.

Thanks, Ned!

To clarify, it's not jus the test suite. OpenSSL 1.1.0c broke the ssl module.

1.1.0d is due today. Who wants to test it? :)