See https://github.com/python/cpython/pull/26104/checks?check_run_id=2662511684

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

The problem could be related to bpo-43921. I neither have a macOS nor a Windows machine to reproduce and debug the issue. Since I'm cannot reproduce the problem on Linux, I'm unable to debug and fix it.

>  I neither have a macOS nor a Windows machine to reproduce and debug the issue.

Can you maybe use a VM for Windows?

I’ll see if I can provoke it on my Mac.

What’s the preferred solution?

1. Catch the connection failure and skip the test?
2. Retry connection?
3. Find the root cause and make sure it never ever happens :)

Maybe 2. is acceptable :)

Is possible that these are flaky tests but I am afraid this may be something more important so o would prefer to go with 3 ;)

I feared that ;)

Getting there. The error message is similar:

$ ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:02:11 load avg: 2.10 [328] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with context.wrap_socket(sock, server_hostname=host) as sslsock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create
    self.do_handshake()
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer

test_ssl failed

== Tests result: FAILURE ==

Got it:

0:02:29 load avg: 2.81 [389] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

test_ssl failed

== Tests result: FAILURE ==

$ git diff
diff --git a/Lib/ssl.py b/Lib/ssl.py
index 2b131de043..9c281d8028 100644
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -257,8 +257,9 @@ class _TLSMessageType:
 if sys.platform == "win32":
     from _ssl import enum_certificates, enum_crls
 
+from test.support import SHORT_TIMEOUT as _GLOBAL_DEFAULT_TIMEOUT
 from socket import socket, SOCK_STREAM, create_connection
-from socket import SOL_SOCKET, SO_TYPE, _GLOBAL_DEFAULT_TIMEOUT
+from socket import SOL_SOCKET, SO_TYPE
 import socket as _socket
 import base64        # for DER-to-PEM translation
 import errno

$ % ./python.exe -m test test_ssl -m test_get_server_certificate -u all -F
...
0:03:27 load avg: 2.24 [535] test_ssl
test test_ssl failed -- Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2132, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2329, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1521, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

test_ssl failed

== Tests result: FAILURE ==

Erlend, could you maybe add some context to the messages, unfortunately, I am failing to understand what they mean :(

me, too. Did the change of import for timeout setting cause the issue?

Yes, sorry ‘bout the cryptic messages :)

This is against current main branch (I’m not at my computer right now, so I don’t have the git ref). The first two failures came after 328 and 389 runs. Increasing the timeout (using test.support.SHORT_TIMEOUT) made it harder trigging the failure (~500 repetitions).

I’ll try to declutter when I’m back on my computer.

New failure in CI: https://github.com/python/cpython/runs/2669949777?check_suite_focus=true

======================================================================
ERROR: test_msg_callback_deadlock_bpo43577 (test.test_ssl.TestSSLDebug)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 4813, in test_msg_callback_deadlock_bpo43577
    s.connect((HOST, server.port))
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1372, in connect
    self._real_connect(addr, False)
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1359, in _real_connect
    super().connect(addr)
ConnectionRefusedError: [Errno 61] Connection refused

Build info for macOS CI attached.

Here's the output when with test.support.verbose = True:

test_ssl: testing with 'OpenSSL 1.1.1k  25 Mar 2021' (1, 1, 1, 11, 15)                                                 
          under Mac ('11.3.1', ('', '', ''), 'x86_64')                                                                 
          HAS_SNI = True                                                                                               
          OP_ALL = 0x80000054                                                                                          
          OP_NO_TLSv1_1 = 0x10000000                                                                                   
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ...  server:  new connection from ('127.0.0.1', 51163)

 server:  bad connection attempt from ('127.0.0.1', 51163):                                                            
Traceback (most recent call last):
   File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2366, in wrap_conn                           
    self.sslconn = self.server.context.wrap_socket(
   File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 518, in wrap_socket                                    
    return self.sslsocket_class._create(
   File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1070, in _create                                       
    self.do_handshake()
   File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1339, in do_handshake                                  
    self._sslobj.do_handshake()
 OSError: [Errno 41] Protocol wrong type for socket                                                                    
ERROR

======================================================================                                                 
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)                                               
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2130, in test_get_server_certificate          
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/test/test_ssl.py", line 2327, in _test_get_server_certificate         
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/src/cpython-ssl/Lib/ssl.py", line 1520, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 844, in create_connection                            
    raise err
  File "/Users/erlendaasland/src/cpython-ssl/Lib/socket.py", line 832, in create_connection                            
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

---------------------------------------------------------------------- 


The mock server fails with EPROTOTYPE, thus the client fails with connection refused.

I found similar issues after doing some googling:
- http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/
- https://github.com/benoitc/gunicorn/issues/1487#issuecomment-333680970

This might be the same macOS bug.

> This might be the same macOS bug.

It could be, but the same thing happens on Windows

> It could be, but the same thing happens on Windows

Are you sure? All the reports are from macOS CI. Christian mentioned that it could be similar to another ssl test failure that also happen on Windows, but those tracebacks do not match the ones in this issue.

See also bpo-43855.

FWIW:

1st shell:
$ ./python.exe -m test test_ssl -u all -m test_get_server_certificate -F
$ ps
[...]
 1271 ttys004    0:01.00 ./python.exe -m test test_ssl -u all -m test_get_server_certificate -F

2nd shell:
$ sudo errinfo -p 1271

The syscall trace shows that errno 41 is returned by write():
      python.exe            write   41  Protocol wrong type for socket 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe          connect   61  Connection refused 
      python.exe          connect   61  Connection refused 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe            ioctl   25  Inappropriate ioctl for device 
      python.exe           stat64    2  No such file or directory

> Are you sure? All the reports are from macOS CI. Christian mentioned that it could be similar to another ssl test failure that also happen on Windows, but those tracebacks do not match the ones in this issue.

I am certainly not sure, but I think they are related.

See #44237 for another repeatedly failing (false positive) ssl test.  These and other randomly failing tests should be skipped for routine CI regression testing of PRs.

Two GHI test macOS failures yesterday, once on main, this is on 3.10 backport
https://github.com/python/cpython/pull/26850/checks?check_run_id=2885797677

ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2180, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/runner/work/cpython/cpython/Lib/test/test_ssl.py", line 2377, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/runner/work/cpython/cpython/Lib/ssl.py", line 1527, in get_server_certificate
    with create_connection(addr, timeout=timeout) as sock:
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 844, in create_connection
    raise err
  File "/Users/runner/work/cpython/cpython/Lib/socket.py", line 832, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused

So far reported (including #44237) ConnectionRefusedError in
test_get_server_certificate X many
test_msg_callback_deadlock_bpo43577
test_get_server_certificate_sni

ConnectionResetError: 
test_connect_cadata

Please, please, skip the randomly failing tests, as is done for other modules.  This has been going on a month now.  It waste other people's time, making merging more painful.

Although I simphatise with the frustration and I would really like to skip them myself, as a release manager I am quite concerned that skip them would masks an actual regression, as these failures are not present in previous versions.

Has these issues (ConnectionRefusedError/ConnectionResetError during handshake) been reported on other systems than macOS? AFAICS, they've only been reported for the macOS CI.

Only on MacOS afaik

See also bpo-33450.

Using this patch, I'm (so far) no longer able to reproduce these test failures:

diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
index cee97a8302..c9c1546467 100644
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -2443,9 +2443,13 @@ def wrap_conn(self):
                 self.server.conn_errors.append(str(e))
                 if self.server.chatty:
                     handle_error("\n server:  bad connection attempt from " + repr(self.addr) + ":\n")
-                self.running = False
-                self.server.stop()
-                self.close()
+
+                # bpo-44229, bpo-43855, bpo-44237, and bpo-33450:
+                # Ignore spurious EPROTOTYPE returned by write() on macOS.
+                if e.errno != errno.EPROTOTYPE and sys.platform != "darwin":
+                    self.running = False
+                    self.server.stop()
+                    self.close()
                 return False
             else:
                 self.server.shared_ciphers.append(self.sslconn.shared_ciphers())



Running for 40 minutes now, right now passing 6k iterations for test_get_server_certificate, test_msg_callback_deadlock_bpo43577, and test_get_server_certificate_sni.

... should perhaps also return True in that case, but it does not seem to have any thing to say.

I can reproduce these issues easily on 3.7, 3.8, and 3.9:

$ python3.7 -m test test_ssl -u all -F -m test_get_server_certificate -v
[...]
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ...  server:  new connection from ('127.0.0.1', 57294)

 server:  bad connection attempt from ('127.0.0.1', 57294):
Traceback (most recent call last):
   File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2313, in wrap_conn
    self.sock, server_side=True)
   File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
   File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
   File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
 OSError: [Errno 41] Protocol wrong type for socket
ERROR

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2108, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Users/erlendaasland/install/lib/python3.7/test/test_ssl.py", line 2273, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Users/erlendaasland/install/lib/python3.7/ssl.py", line 1313, in get_server_certificate
    with  create_connection(addr) as sock:
  File "/Users/erlendaasland/install/lib/python3.7/socket.py", line 728, in create_connection
    raise err
  File "/Users/erlendaasland/install/lib/python3.7/socket.py", line 716, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 61] Connection refused


$ python3.8 -m test test_ssl -u all -F -m test_get_server_certificate -v
test_ssl: testing with 'OpenSSL 1.1.1k  25 Mar 2021' (1, 1, 1, 11, 15)
          under Mac ('10.16', ('', '', ''), 'x86_64')
          HAS_SNI = True
          OP_ALL = 0x80000054
          OP_NO_TLSv1_1 = 0x10000000
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ...  server:  new connection from ('127.0.0.1', 59370)

 server:  bad connection attempt from ('127.0.0.1', 59370):
Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2348, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
   File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
 OSError: [Errno 41] Protocol wrong type for socket
ERROR

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2142, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/test/test_ssl.py", line 2309, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1484, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer


$ python3.9 -m test test_ssl -u all -F -m test_get_server_certificate -v
[...]
test_ssl: testing with 'OpenSSL 1.1.1k  25 Mar 2021' (1, 1, 1, 11, 15)
          under Mac ('10.16', ('', '', ''), 'x86_64')
          HAS_SNI = True
          OP_ALL = 0x80000054
          OP_NO_TLSv1_1 = 0x10000000
test_get_server_certificate (test.test_ssl.SimpleBackgroundTests) ...  server:  new connection from ('127.0.0.1', 60096)

 server:  bad connection attempt from ('127.0.0.1', 60096):
Traceback (most recent call last):
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2353, in wrap_conn
    self.sslconn = self.server.context.wrap_socket(
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
 OSError: [Errno 41] Protocol wrong type for socket
ERROR

======================================================================
ERROR: test_get_server_certificate (test.test_ssl.SimpleBackgroundTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2147, in test_get_server_certificate
    _test_get_server_certificate(self, *self.server_addr, cert=SIGNING_CA)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/test/test_ssl.py", line 2314, in _test_get_server_certificate
    pem = ssl.get_server_certificate((host, port), ca_certs=cert)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1484, in get_server_certificate
    with context.wrap_socket(sock) as sslsock:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
ConnectionResetError: [Errno 54] Connection reset by peer


It does not seem to be specific to 3.10/3.11.

New changeset b5a52eef67997246b4235b5407e52a01e822ce56 by Erlend Egeberg Aasland in branch 'main':
bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
https://github.com/python/cpython/commit/b5a52eef67997246b4235b5407e52a01e822ce56

New changeset 0796e21fea31fe7b697d84c8d03186817792c458 by Miss Islington (bot) in branch '3.9':
bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
https://github.com/python/cpython/commit/0796e21fea31fe7b697d84c8d03186817792c458

New changeset b3fac2926b23b4f1342099e591aa3fed7f16876d by Miss Islington (bot) in branch '3.10':
bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893)
https://github.com/python/cpython/commit/b3fac2926b23b4f1342099e591aa3fed7f16876d

Marking bpo-43855 as a duplicate of this issue.

Marking bpo-44237 as a duplicate of this issue.

FYI, here's a CI failure against 3.9:
https://github.com/python/cpython/runs/2894813367?check_suite_focus=true

New changeset 71ba16b21cb35923098026117b5e6d823c5f5707 by Miss Islington (bot) in branch '3.8':
bpo-44229: Ignore spurious EPROTOTYPE on macOS in test_ssl (GH-26893) (GH-26895)
https://github.com/python/cpython/commit/71ba16b21cb35923098026117b5e6d823c5f5707

I haven't observed this issue on the CI since GH-26893 through GH-26896 was merged. Marking this as resolved. If anyone disagrees, feel free to reopen :)

http://erickt.github.io/blog/2014/11/19/adventures-in-debugging-a-potential-osx-kernel-bug/ says:

"If we trigger a send while the kernel is in the middle of tearing down the socket, it returns EPROTOTYPE." (instead of ECONNRESET)

Maybe send() could raise a ConnectionResetError exception on EPROTOTYPE?

But here, it is write() that returns EPROTOTYPE. See msg394798.

I'd be interested in hearing Ronald's opinion. (Added to nosy.)