This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: socket.sendall() cannot send buffers of 2GB or more
Type: behavior Stage: resolved
Components: Extension Modules Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Tom.van.Leeuwen, amaury.forgeotdarc, christian.heimes, giampaolo.rodola, martin.panter, noxiouz, pitrou, vstinner
Priority: normal Keywords: patch

Created on 2013-05-30 12:56 by Tom.van.Leeuwen, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
send_and_sendall_issue18100.patch noxiouz, 2013-07-07 20:48 change type fpr len from int to size_t review
Messages (13)
msg190357 - (view) Author: Tom van Leeuwen (Tom.van.Leeuwen) Date: 2013-05-30 12:56
When using socket.sendall() to send a buffer of 2GB or more (for example, numpy.zeros(2*GB, dtype=numpy.uint8) ), it doesn't send anything at all.

In socketmodule.c, socket.sendall() sock_sendall uses an int for len, while this should be a Py_ssize_t.
msg190361 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2013-05-30 14:48
"int len = pbuf.len;"
Why doesn't gcc emit a warning here?
msg192588 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-07 20:48
The same error in the use of socket.send(). Is it possible to use size_t for len instead of int?
According to http://www.python.org/dev/peps/pep-0353/ Py_ssize_t on x86 is typedef for int, and size_t has the same size.
On x64 sizeof size_t is 8bit, that covers the limits of reasonable use.
msg192593 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-07 21:05
typo fix
8bite -> 8 bytes
msg192698 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-08 21:03
This issue is similar like http://bugs.python.org/issue9566.
It seems reasonable to apply a similar fix.

But why not use the types described in the signature functions in <sys/socket.h>. In particular use for len size_t, and for n - ssize_t?
msg192699 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-07-08 22:24
Does it affect Python 3, too?
msg192732 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-09 09:18
There seems to be no, because there's this bug is already fixed in Python 3, according to http://hg.python.org/cpython/file/c1a400501db6/Modules/socketmodule.c#l3290. 
But the use of Py_ssize_t (len and n) hides the potential type conversion.
msg192734 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-07-09 09:44
> But the use of Py_ssize_t (len and n) hides the potential type conversion.

Which type conversion?
msg192735 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-09 09:50
I could be wrong, but:

ssize_t
send(int socket, const void *buffer, size_t length, int flags);

so length is size_t, but Py_ssize_t is ssize_t (signed to unsigned).

PS. sorry for my bad English
msg192778 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-07-09 20:52
> ssize_t send(int socket, const void *buffer, size_t length, int flags);

Oh, I didn't notice that. I hope that the result always fit in a ssize_t. If it is not the case, it is probably a bug in the kernel. An example of such bug:
http://xorl.wordpress.com/2009/04/10/cve-2009-1265-linux-kernel-rosex25netrom-integer-overflows/

On Solaris, read(), send() and probably other functions return EINVAL if input length argument overflows a ssize_t. We may be extra-safe by truncating the length to PY_SSIZE_T_MAX. But I would appreciate a test on Linux and Solaris with such huge values. Which kind of socket should be used to test such buffer?
msg192937 - (view) Author: Anton Tyurin (noxiouz) Date: 2013-07-12 12:31
According to man send: 
only sendmsg() if input length argument overflows a ssize_t on OS X. But truncating extradata in sendall() without exception is bad idea, because sendall() never returns count of successfully sent bytes. So we'll never know about incomplete data transmition.

Is it good idea to test such buffer on Unix socket, for example?
msg266040 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-05-22 01:42
Looks like this was fixed in 3.1 and 3.2 by r84150.
msg404446 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-10-20 13:15
The issue is fixed in Python 3 by commit f72006f4429975a9d221e046e43dabd4f41eda23
History
Date User Action Args
2022-04-11 14:57:46adminsetgithub: 62300
2021-10-20 13:15:17christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg404446

stage: patch review -> resolved
2016-05-22 01:42:23martin.pantersetnosy: + martin.panter

messages: + msg266040
stage: needs patch -> patch review
2013-08-31 19:04:46giampaolo.rodolasetnosy: + giampaolo.rodola
2013-07-12 12:31:27noxiouzsetmessages: + msg192937
2013-07-09 20:52:00vstinnersetmessages: + msg192778
2013-07-09 09:50:43noxiouzsetmessages: + msg192735
2013-07-09 09:44:05vstinnersetmessages: + msg192734
2013-07-09 09:18:22noxiouzsetmessages: + msg192732
2013-07-08 22:24:21christian.heimessetnosy: + christian.heimes
messages: + msg192699
2013-07-08 21:03:05noxiouzsetmessages: + msg192698
2013-07-07 21:06:00vstinnersetnosy: + vstinner
2013-07-07 21:05:16noxiouzsetmessages: + msg192593
2013-07-07 20:48:28noxiouzsetfiles: + send_and_sendall_issue18100.patch

nosy: + noxiouz
messages: + msg192588

keywords: + patch
2013-05-30 14:48:48amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg190361
2013-05-30 14:22:30serhiy.storchakasetnosy: + pitrou
stage: needs patch

components: + Extension Modules, - Build
versions: + Python 2.7, - Python 2.6
2013-05-30 12:56:22Tom.van.Leeuwencreate