Due to some rumblings over on the mentors list and python-dev, this is now getting some love.

Guido has stated that something should make it into the subprocess module for 3.5 in this email: https://groups.google.com/d/msg/dev-python/I6adJLIjNHk/Usrvxe_PVJIJ

His suggested API is:
proc.write_nonblocking(data)
data = proc.read_nonblocking()
data = proc.read_stderr_nonblocking()


I've implemented the API for Windows and Linux, and below are my findings.


On the Linux side of things, everything seems to be working more or less as expected, though in writing tests I did need to add an optional argument to qcat.py in order to have it flush its stdout to be able to read from the parent process. Odd, but whatever.

Also, Richard Oudkerk's claims above about not needing to use fcntl to swap flags is not correct. It's necessary to not block on reading, even if select is used. Select just guarantees that there is at least 1 byte or a closed handle, not that your full read will complete.


On the Windows side of things, my assumptions about WriteFile() were flawed, and it seems that the only way to make it actually not block if/when the outgoing buffer is full is to create a secondary thread to handle the writing*. I've scoured the MSDN docs and there doesn't seem to be an available API for interrogating the pipe to determine whether the buffer is full, how full it is, or whether the write that you'd like to do will succeed or block.

* This applies even with the use of asyncio. Because the intent for the call is to return more or less immediately, a thread still has to be created to handle the event loop for IO completion, which means that asyncio can't prevent the use of threads and not block without basically integrating an event loop into the caller.

Considering that the Windows communicate() method already uses threads to handle reading and writing, I don't believe it is a big deal to add it for this situation too. Any major objections?