classification
Title: socket loses data, calling send()/sendall() on invalid socket does not report error and returns all bytes as written
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.1
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: diekmann, neologix, pitrou
Priority: normal Keywords:

Created on 2010-12-07 09:01 by diekmann, last changed 2010-12-10 17:26 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
socketbug.py diekmann, 2010-12-07 09:01 reproduce issue
Messages (9)
msg123534 - (view) Author: diekmann (diekmann) Date: 2010-12-07 09:01
Calling send()/sendall() on an invalid socket does not report an error and returns all bytes as written. Thus, all bytes written to the socket are lost and the application is not informed about that and treats the bytes as successfully sent. The bytes given to the socket are silently lost, this cannot be recovered.

The attached file includes an example to reproduce this problem. I defined an invalid socket, when the other side of the connection has closed the connection.

Steps to reproduce (see attached file for python implementation):
1) Create listening socket
2) let client connect to it
2.1) send something to the client (optional step)
3) Client terminates connection (now the socket on the server side is invalid)
4) Server calls send/sendall <--- No Error here, but everything is lost
5) Server calls send/sendall again (Now we get the required error)
msg123548 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-07 13:46
Which OS is it? It works for me:

Traceback (most recent call last):
  File "socketbug.py", line 24, in <module>
    print(con.send(bytes("Hello", "ascii")))
socket.error: [Errno 32] Broken pipe

Regardless, the error returned is the one returned by your OS here. If your OS accepts the send(), Python can nothing against it.
msg123565 - (view) Author: diekmann (diekmann) Date: 2010-12-07 17:14
ubuntu 9.10 with python 3.1.1+ and debian 5.0.6 with Python 3.1.3rc1

I can reproduce the bug on both systems. Maybe it has been fixed in python 3.2?
msg123567 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-07 17:37
I tried on both 3.1.3 and 3.2. It works (raises an error) under Mandriva; I've just tested under Debian stable and it fails.

Looking at netstat, the difference seems to be that "nc" closes the TCP connection fine under Mandriva when killed, and doesn't under Debian. Again, Python only propagates whatever error code the OS does or doesn't return.
msg123597 - (view) Author: diekmann (diekmann) Date: 2010-12-08 10:17
The Documentation states:

socket.sendall(bytes[, flags])¶
    Send data to the socket. The socket must be connected to a remote socket. The optional flags argument has the same meaning as for recv() above. Unlike send(), this method continues to send data from bytes until either all data has been sent or an error occurs. None is returned on success. On error, an exception is raised, and there is no way to determine how much data, if any, was successfully sent.

This is not consistent with the results reproduced above, however, the results from above are exactly what should happen. Maybe there should be a remark, that the return value of sendall (and send) may be system dependent. Or a patch which enforces the documented behviour of sendall, regardless of the operating system would be a nice future feature?
msg123599 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-08 10:51
> This is not consistent with the results reproduced above, however, the
> results from above are exactly what should happen. Maybe there should
> be a remark, that the return value of sendall (and send) may be system
> dependent.

Pretty much all the socket module is system-dependent, and it is
actually stated at the top of the documentation:

“Some behavior may be platform dependent, since calls are made to the
operating system socket APIs.”
http://docs.python.org/dev/library/socket.html

> Or a patch which enforces the documented behviour of sendall,
> regardless of the operating system would be a nice future feature?

How could it work?
msg123600 - (view) Author: diekmann (diekmann) Date: 2010-12-08 12:23
> How could it work?
Before sending the actual data to the socket, send an empty packet to the socket and check if it is still alive. This may be a large performance issue on a lower level (if the connection is TCP, we want to wait for the ACK), but on a higher level, when using sendall() with larger arguments, it may be very convenient and the performance overhead may be barely noticeable.
I guess when using high-level functions like sendall(), the bare funcionality is preferred over this performance issue.
msg123632 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2010-12-08 18:51
>  it may be very convenient and the performance overhead may be barely noticeable.

Convenient for what ?
If the remote end doesn't send a FIN or RST packet, then the TCP/IP stack has no way of knowing the remote end is down.
Successfull return of send(2) never meant a succesfull delivery to the other end, see man page :
"No indication of failure to deliver is implicit in a send(). Locally detected errors are indicated by a return value of -1. "

If your remote application doesn't close its socket cleanly, then your application is broken.
To guard against that, you could use TCP keepalive...
msg123739 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-12-10 17:26
Ok, closing as invalid.
History
Date User Action Args
2010-12-10 17:26:37pitrousetstatus: open -> closed
resolution: not a bug
messages: + msg123739
2010-12-08 18:51:55neologixsetnosy: + neologix
messages: + msg123632
2010-12-08 12:23:55diekmannsetmessages: + msg123600
2010-12-08 10:51:00pitrousetmessages: + msg123599
2010-12-08 10:17:06diekmannsetmessages: + msg123597
2010-12-07 17:37:13pitrousetmessages: + msg123567
2010-12-07 17:14:21diekmannsetstatus: pending -> open
resolution: not a bug -> (no value)
messages: + msg123565

versions: - Python 3.2
2010-12-07 13:46:33pitrousetstatus: open -> pending
versions: + Python 3.2
nosy: + pitrou

messages: + msg123548

resolution: not a bug
2010-12-07 09:01:24diekmanncreate