classification
Title: non-blocking SSL write fails if a partial write was issued
Type: enhancement Stage: resolved
Components: Extension Modules, SSL, Windows Versions: Python 3.7, Python 3.6
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, dsiroky, giampaolo.rodola, jcea, paul.moore, pitrou, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2011-05-27 18:06 by dsiroky, last changed 2017-09-07 02:10 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
ssl_sock_test_working_but_blocking.py dsiroky, 2011-05-27 18:06 example to reproduce (with repeated writes, this works well)
ssl_sock_test_fix.py dsiroky, 2011-06-08 09:24 example to reproduce
Messages (7)
msg137092 - (view) Author: David Siroky (dsiroky) Date: 2011-05-27 18:06
Trying to send large bulk of data in MS Windows via non-blocking SSLSocket raises an exception but part of the data is delivered.

E.g.

ssl_socket.write(b"a" * 200000)

raises

ssl.SSLError: [Errno 3] _ssl.c:1126: The operation did not complete (write)

There is no way to get the sent bytes count which is essential in non-blocking communication.

ssl_socket.send() returns 0. This should be unified with posix behavior.
msg137810 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-06-07 11:36
Which version of Python are you testing on? It works fine using 3.2 and 3.3 here, under Windows 7 64-bit.

Anyway, I would suggest to batch your write in smaller chunks (say, 2048 bytes each). Also, you may try sendall() instead.
msg137898 - (view) Author: David Siroky (dsiroky) Date: 2011-06-08 09:24
Sorry, I attached wrong example version. It uses repeated sslsock.write() of the same buffer after catching SSL_ERROR_WANT_WRITE. It delivers the full block but this is a blocking operation.

I'm troubled with non-blocking writes. But as I dig deeper into the problem it looks like an inconsistency in OpenSSL (Linux vs. MSW). In Linux sslsock.write() always (as far as I can tell) sends some or all of the data and returns the amount. Like the plain socket.send(). In Windows it raises an exception if the data is larger then some uncertain size (mostly 16kB).

I'm working on snakeMQ asynchronous messaging library where the "packeter" layer relies on the sock.send() returning actual amount of sent bytes. This is not working in Windows if the link uses SSL.

Tested on WinXP, py2.6, py3.2.
msg137916 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-06-08 16:00
> Sorry, I attached wrong example version. It uses repeated
> sslsock.write() of the same buffer after catching
> SSL_ERROR_WANT_WRITE. It delivers the full block but this is a
> blocking operation.

In "normal" non-blocking code you would use select() (or an equivalent)
until the socket is ready for writing, and then send() some chunk of
data on it. Can't you use that approach?
msg137948 - (view) Author: David Siroky (dsiroky) Date: 2011-06-09 11:18
I didn't meant blocking as setblocking(True). I use select/poll but I can't use returned value from send() immediately since in Windows there are often needed more send rounds to actually know how much data was sent.

E.g. in Linux I know it after the first call:

  sslsock.write("abcd") -> returns 2
  poll(sslsock)
  sslsock.write("cd")

in Windows I must do:

  sslsock.write("abcd") -> raises SSLError
  poll(sslsock)
  sslsock.write("abcd") -> returns 4

As I wrote it might be inconsistency in OpenSSL and not in Python's wrapper.
msg138116 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-06-10 17:11
I don't see any discrepancy here (with Python 3.3). Under both Linux and Windows, the client thread prints:

ERR [Errno 3] _ssl.c:1126: The operation did not complete (write)
done

The only difference is that the server thread receives 128KB under Linux, and 48KB under Windows.

I think the main variation comes not from OpenSSL, but from the OS's thread scheduling algorithms. If you do a large write, you have to be prepared to get an error, because the buffers may be too small.

I agree it might seem strange that OpenSSL returns an error rather than the number of sent bytes, and this behaviour could actually be changed at the OpenSSL level (see http://www.openssl.org/docs/ssl/SSL_write.html) by allowing people to use the SSL_MODE_ENABLE_PARTIAL_WRITE option. However, doing so by default could break existing code which assumes that writes either fail or succeed completely.

issue8240 talks about a slightly related request.
msg301561 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2017-09-07 02:10
I'm closing this issue because it hasn't seen any activity in six years. Please reopen it with more information if your problem still persists.
History
Date User Action Args
2017-09-07 02:10:35christian.heimessetstatus: open -> closed
messages: + msg301561

assignee: christian.heimes ->
resolution: out of date
stage: needs patch -> resolved
2016-09-15 07:50:05christian.heimessetassignee: christian.heimes

components: + SSL
nosy: + paul.moore, tim.golden, zach.ware, steve.dower
2016-09-08 22:57:58christian.heimessetstage: needs patch
versions: + Python 3.6, Python 3.7, - Python 3.5
2014-07-30 22:02:54BreamoreBoysetversions: + Python 3.5, - Python 3.3
2013-06-17 18:23:00pitrousetnosy: + christian.heimes
2011-06-16 14:22:51jceasetnosy: + jcea
2011-06-10 17:11:12pitrousettype: behavior -> enhancement
title: non-blocking SSL write in Windows sends large data but raises exception -> non-blocking SSL write fails if a partial write was issued
messages: + msg138116
versions: + Python 3.3, - Python 2.6, Python 3.2
2011-06-09 11:18:04dsirokysetmessages: + msg137948
2011-06-08 18:52:14giampaolo.rodolasetnosy: + giampaolo.rodola
2011-06-08 16:00:38pitrousetmessages: + msg137916
2011-06-08 09:24:32dsirokysetfiles: + ssl_sock_test_fix.py

messages: + msg137898
2011-06-07 11:36:41pitrousetnosy: + pitrou
messages: + msg137810
2011-05-27 18:06:48dsirokycreate