classification
Title: Add utilities to get/set pipe and socket buffer sizes?
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.8
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: njs, pitrou, vstinner, yselivanov
Priority: normal Keywords:

Created on 2018-06-01 13:18 by vstinner, last changed 2018-06-20 10:13 by vstinner. This issue is now closed.

Messages (3)
msg318411 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-01 13:18
Many Python unit tests require a blocking send into a pipe or a socket.

support.PIPE_MAX_SIZE (4 MiB +1 B) has been added for pipes in bpo-17835: sending PIPE_MAX_SIZE into a pipe must block.

On Linux, the maximum size of a pipe is /proc/sys/fs/pipe-max-size. Since Linux 2.6.35, it's now possible to get and set the size of a pipe using fcntl with F_GETPIPE_SZ and F_SETPIPE_SZ commands.

For sockets, support.SOCK_MAX_SIZE (16 MiB) has been added in bpo-18643. It's possible to get/set the size of receive and send socket buffers using getsockopt() and setsockopt() with SO_RCVBUF and SO_SNDBUF commands.

For pipes, I'm not sure that it's possible to get the size of a pipe buffer in a portable way. For example, F_GETPIPE_SZ was only introduced in Linux 2.6.35.

Since the first user of these features are tests, maybe we can start with best-effort functions in test.support.

Recently, I got issues with buffer sizes in test_multiprocessing_forkserver.test_ignore() (bpo-33532) and sendfile tests of test_asyncio (bpo-33353).
msg318428 - (view) Author: Nathaniel Smith (njs) * (Python committer) Date: 2018-06-01 15:54
Note that in my experience, socket systems treat the buffer sizes as more like... rough guidelines. Especially Windows and Linux. Which doesn't mean they're not useful to expose somehow, but you can't assume that just because you set the buffer to size X means that operations will start blocking after X bytes. When trio's tests need a clogged socket, we just keep sending in a loop until we observe a BlockingIOError, and then do the test.

Pipe buffers are much more reasonably behaved IME.
msg320052 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-06-20 10:13
> Note that in my experience, socket systems treat the buffer sizes as more like... rough guidelines.

Oh. Now I'm scared.

I succeeded to fix bpo-33532 and bpo-33353, so I no longer need this issue.

Since it seems hard to have a portable and *reliable* behaviour, I prefer to abandon this idea. Sorry!
History
Date User Action Args
2018-06-20 10:13:51vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg320052

stage: resolved
2018-06-01 15:54:39njssetmessages: + msg318428
2018-06-01 13:18:56yselivanovsetnosy: + njs
2018-06-01 13:18:19vstinnercreate