While writing tests for nntplib, it came to me that a PipeIO or BytesPipeIO would be useful (I actually wrote a minimal one). It would be a non-seekable in-memory bytes buffer with distinct read and write pointers, so as to act like a system FIFO or a socket.makefile() object. A pure Python implementation would probably be sufficient, since the main use would be for test purposes. What do you think?

(you may point to os.pipe() but it has problems such as limited buffer size: try to write many bytes to the write end and it will block until the other end tries to read something)

> It would be a non-seekable in-memory bytes buffer with distinct 
> read and write pointers, so as to act like a system FIFO or a 
> socket.makefile() object.

What would it do when the equivalent system FIFO object would block?

> (you may point to os.pipe() but it has problems such as limited buffer
> size: try to write many bytes to the write end and it will block until
> the other end tries to read something)

In the past, I've gotten around that limitation by using threads (which, admittedly, certainly had its own drawbacks)

> > It would be a non-seekable in-memory bytes buffer with distinct 
> > read and write pointers, so as to act like a system FIFO or a 
> > socket.makefile() object.
> 
> What would it do when the equivalent system FIFO object would block?

Simply buffer everything.
(this is more aimed at mocking sockets than system FIFOs; system FIFOs
can be trivially wrapped in a BufferedRWPair)

> > (you may point to os.pipe() but it has problems such as limited buffer
> > size: try to write many bytes to the write end and it will block until
> > the other end tries to read something)
> 
> In the past, I've gotten around that limitation by using threads
> (which, admittedly, certainly had its own drawbacks)

Sure, but that's a lot of complication in itself.

The _multiprocessing module already have similar objects.
_multiprocessing.Connection is based on file descriptors (and initialized with the result of os.pipe())
_multiprocessing.PipeConnection uses Windows named pipes.

> Simply buffer everything.

That works for when a write operation would block.  What would it do when a read operation would block?

> > Simply buffer everything.
> 
> That works for when a write operation would block.  What would it do
> when a read operation would block?

Good question. It could either return an empty bytes object, or perhaps
raise EOFError (it could be a constructor argument).

> The _multiprocessing module already have similar objects.
> _multiprocessing.Connection is based on file descriptors (and initialized with the result of os.pipe())
> _multiprocessing.PipeConnection uses Windows named pipes.

_multiprocessing.Connection exposes a socket-like API (recv/send) rather
than file-like (read/write/etc.).
Furthermore, _multiprocessing isn't available everywhere; and I wouldn't
really trust it in terms of stability.

Another option would be to have the read-end act like a non-blocking socket (i.e., raise EAGAIN).

Since it would mostly be for testing, would it make more sense to add it to test.support or someplace similar instead of io?

> Another option would be to have the read-end act like a non-blocking
> socket (i.e., raise EAGAIN).
> 
> Since it would mostly be for testing, would it make more sense to add
> it to test.support or someplace similar instead of io?

I thought it might be useful for third-party libraries, or even
non-testing situations (although obviously the latter wouldn't be very
common). But, yes, a starting point could be test.support. It would also
avoid having to write unit tests and docs :)
In this case, I would like to do it as a part of the nntplib patch.

> I thought it might be useful for third-party libraries, or even
> non-testing situations 

That might be true, but I prefer that we give code a more vigorous exercise before putting it in the standard library proper.  When we try to use it for something more than nntplib tests we might find that we need to alter its behavior slightly or flesh out the functionality more.

> But, yes, a starting point could be test.support. It would also
> avoid having to write unit tests and docs :)

+1 from me.