classification
Title: race condition between send and recv in _ssl with non-zero timeout
Type: enhancement Stage: needs patch
Components: Documentation, SSL Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: christian.heimes, docs@python, ned.deily, nneonneo, ronaldoussoren, tacocat
Priority: normal Keywords:

Created on 2018-03-16 19:43 by nneonneo, last changed 2020-10-23 09:37 by christian.heimes.

Files
File name Uploaded Description Edit
test_ssl.py nneonneo, 2018-03-16 19:43 Python script to demonstrate the issue
Messages (4)
msg313970 - (view) Author: Robert Xiao (nneonneo) * Date: 2018-03-16 19:43
Environment: Several versions of Python (see below), macOS 10.12.6

The attached script creates an SSL echo server (fairly standard), connects to the server, and spawns a read and write thread.

The write thread repeatedly shovels data into the connection, while the read thread receives data and prints a dot for each successful read.

The socket has a timeout of 10 seconds set: if the timeout is 0, the script blows up immediately due to blocking, and if the timeout is -1 nothing bad happens.

On Linux and the default Mac Python 2.6, the script prints an endless series of dots as expected.

On most other versions of Mac Python (2.7, 3.5, 3.6), the script dies quite quickly (within 1-2 seconds) with an error like this:


$ /usr/bin/python2.7 test_ssl.py 
Got connection from ('127.0.0.1', 49683)
..................................Exception in thread ReadThread:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test_ssl.py", line 93, in read_thread
    csocket.recv()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 734, in recv
    return self.read(buflen)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 621, in read
    v = self._sslobj.read(len or 1024)
error: [Errno 35] Resource temporarily unavailable


The error can be one of the following:

[Py2.7] error: [Errno 35] Resource temporarily unavailable
[Py2.7] SSLWantReadError: The operation did not complete (read) (_ssl.c:1752)
[Py3.x] BlockingIOError: [Errno 35] Resource temporarily unavailable
[Py3.x] ssl.SSLWantReadError: The operation did not complete (read) (_ssl.c:1974)
[Py3.6] ssl.SSLError: Invalid error code (_ssl.c:2217)

The last error occurs under much rarer circumstances, but appears to be associated with the same underlying bug. The "invalid error code" is 0 when tested with a debugger, indicating a successful completion (but somehow the error logic gets triggered anyway).

This was tested with the following configurations:

macOS: /usr/bin/python2.6: Python 2.6.9 from Apple [ok]
macOS: /usr/bin/python2.7: Python 2.7.10 from Apple [crashes]
macOS: /usr/local/bin/python2.7: Python 2.7.13 from Python.org [crashes]
macOS: /usr/local/bin/python3.5: Python 3.5.1 from Python.org [crashes]
macOS: /usr/local/bin/python3.6: Python 3.6.4 from Python.org [crashes]
macOS: /opt/local/bin/python2.7: Python 2.7.14 from MacPorts [crashes]

A number of these were tested on a second machine (to rule out any strange environment issues), and the same results were obtained.
msg379420 - (view) Author: Ronald Oussoren (ronaldoussoren) * (Python committer) Date: 2020-10-23 09:12
The script still fails on macOS 10.15 with python 3.9 (installed from python.org).

Interestingly enough:
- First run of the script worked without any problems
- Subsequent runs failed with various errors:

$ python3.9 t.py
Got connection from ('127.0.0.1', 60640)
Exception in thread ReadThread:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 93, in read_thread
    csocket.recv()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1226, in recv
    return self.read(buflen)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
    return self._sslobj.read(len)
BlockingIOError: [Errno 35] Resource temporarily unavailable


----

Got connection from ('127.0.0.1', 60651)
.Traceback (most recent call last):
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 74, in server_thread
Exception in thread WriteThread:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 950, in _bootstrap_inner
Exception in thread ReadThread:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 950, in _bootstrap_inner
    deal_with_client(newsocket)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 68, in deal_with_client
    data = connstream.read()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 888, in run
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 888, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 93, in read_thread
    self._target(*self._args, **self._kwargs)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 89, in write_thread
    return self._sslobj.read(len)
    csocket.recv()
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2621)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1226, in recv
    csocket.send(b'blah')
Closed connection from ('127.0.0.1', 60651)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1173, in send
    return self.read(buflen)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
    return self._sslobj.write(data)
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:2471)
    return self._sslobj.read(len)
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_RECORD_MAC] sslv3 alert bad record mac (_ssl.c:2621)

----

Got connection from ('127.0.0.1', 60636)
.Exception in thread ReadThread:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 950, in _bootstrap_inner
Traceback (most recent call last):
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 74, in server_thread
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 888, in run
    deal_with_client(newsocket)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 68, in deal_with_client
    self._target(*self._args, **self._kwargs)
  File "/Users/ronald/Projects/python/github/cpython-ronald/t.py", line 93, in read_thread
    csocket.recv()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1226, in recv
    data = connstream.read()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
    return self.read(buflen)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/ssl.py", line 1101, in read
    return self._sslobj.read(len)
    return self._sslobj.read(len)
ssl.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2621)
Closed connection from ('127.0.0.1', 60636)
BlockingIOError: [Errno 35] Resource temporarily unavailable
Exception in thread WriteThread:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 950, in _bootstrap_inner
msg379423 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-23 09:36
The demo script uses the same connection socket in two threads. Neither OpenSSL nor Python guarantee that a single SSLSocket object can be used by multiple threads. https://www.openssl.org/blog/blog/2017/02/21/threads/ contains more information about thread safety.

tl;dr SSLObject and SSLSocket cannot be safely used by multiple threads at the same time. SSLContext can be shared across multiple threads. However it's not safe to reconfigure a context once it's attached to a connection. Only getters, wrap_*(), and load_verify_locations() are safe.
msg379424 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-10-23 09:37
I'm turning this into a docs bug.
History
Date User Action Args
2020-10-23 09:37:06christian.heimessetassignee: christian.heimes -> docs@python
type: crash -> enhancement
components: + Documentation, - macOS
versions: + Python 3.8, Python 3.9, Python 3.10, - Python 2.7, Python 3.5, Python 3.6
nosy: + docs@python

messages: + msg379424
stage: needs patch
2020-10-23 09:36:08christian.heimessetmessages: + msg379423
2020-10-23 09:12:37ronaldoussorensetmessages: + msg379420
2018-04-17 21:18:45tacocatsetnosy: + tacocat
2018-03-16 19:43:10nneonneocreate