Presumuing your file descriptor 3 is the read end of the pipe to the child’s output, then there is probably a process somewhere that could still write to the write end. Normally “check_output” waits until it has read all possible output from the pipe(s).

This is probably not a bug in Python. Maybe it is a bug with SSH or your “MUX” (you didn’t explain what that is) leaving a process running in the background that could output to stderr. Try to track down what processes have your pipe open. Find out the number that identifies the pipe. It is the node number in the “lsof” command, or in the symbolic link under /proc:

$ lsof -a -c python -d 3
COMMAND   PID    USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python3 26025 vadmium    3r  FIFO    0,8      0t0 4913217 pipe
$ ls -l "/proc/$(pgrep python)/fd/3"
lr-x------ 1 vadmium vadmium 64 Oct  6 22:17 /proc/26025/fd/3 -> pipe:[4913217]

Then look through the other files to find if other process(es) have the write end of the pipe open; “cat” in my example:

$ lsof | grep 4913217
python3   26025       vadmium    3r     FIFO        0,8      0t0    4913217 pipe
cat       26026       vadmium    1w     FIFO        0,8      0t0    4913217 pipe
$ ls -l /proc/*/fd/* | grep 4913217
lr-x------ 1 vadmium vadmium 64 Oct  6 22:17 /proc/26025/fd/3 -> pipe:[4913217]
l-wx------ 1 vadmium vadmium 64 Oct  6 22:16 /proc/26026/fd/1 -> pipe:[4913217]

The general problem does seem to be a recurring thing with the subprocess module, so maybe more documentation or other enhancements could help. Similar reports:

* Issue 31447: communicate not respecting timeout due to grandchild process
* Issue 30154: subprocess.run with timeout and output pipe from grandchild
* Issue 26534: check_output with shell=True and timeout doesn’t kill shell’s child
* Issue 23213: communicate hang with stderr leaked in “systemd” daemon
* Issue 13422: communicate hangs as long as a daemon leaves pipes open
* Issue 4216: communicate hang after starting “cpboot” service