Windows 10 1709 x64, Python 3.7.1.

Minimal example and stack traces at https://gist.github.com/jimbo1qaz/75d7a40cac307f8239ce011fd90c86bf

Essentially I create a subprocess.Popen, using a process (msys2 head.exe) which closes its stdin after some amount of input, then write nothing but b"\n"*1000 bytes to its stdin.

If the bufsize is small (1000 bytes), I always get OSError: [Errno 22] Invalid argument

If the bufsize is large (1 million bytes), I always get BrokenPipeError: [Errno 32] Broken pipe. (This happens whether I write 1 million newlines or 1000 at a time).

Originally I created a ffmpeg->ffplay pipeline with a massive bufsize (around 1280*720*3 * 2 frames), then wrote 1280*720*3 bytes of video frames at a time. Closing ffplay's window usually created BrokenPipeError, but occasionally OSError. This was actually random.


------------

It seems that this is known to some extent, although I couldn't find any relevant issues on the bug tracker, and "having to catch 2 separate errors" isn't explained on the documentation. (Is it intended though undocumented behavior?)

Popen._communicate() calls Popen._stdin_write(), but specifically ignores BrokenPipeError and OSError where exc.errno == errno.EINVAL == 22 (the 2 cases I encountered).

But I don't call Popen.communicate() but instead write directly to stdin, since I have a loop that outputs 1 video frame at a time, and rely on pipe blocking to stop my application from running too far ahead of ffmpeg/ffplay.

------------

popen.stdin is a <_io.BufferedWriter name=3>.

https://docs.python.org/3/library/io.html#io.BufferedIOBase.write

>Write the given bytes-like object, b, and return the number of bytes written (always equal to the length of b in bytes, since if the write fails an OSError will be raised). Depending on the actual implementation, these bytes may be readily written to the underlying stream, or held in a buffer for performance and latency reasons.

The page doesn't mention BrokenPipeError at all (Ctrl+F). So why do I *sometimes* get a BrokenPipeError (subclasses ConnectionError subclasses OSError) instead?

Popen._stdin_write() has comments linking to https://bugs.python.org/issue19612 and https://bugs.python.org/issue30418 .

If I'm right, we can reduce your example down as follows:

    import os
    import subprocess
    import time
    import ctypes

    RtlGetLastNtStatus = ctypes.WinDLL('ntdll').RtlGetLastNtStatus
    RtlGetLastNtStatus.restype = ctypes.c_ulong

    msys = os.path.normpath("C:/msys64/usr/bin")
    head = os.path.join(msys, "head.exe")

    p = subprocess.Popen(head, stdin=subprocess.PIPE,
            stdout=subprocess.PIPE, bufsize=0)

    # head.exe reads 1 KiB. It closes stdin if it finds 10 lines.
    p.stdin.write(b'\n' * 1024)

    # If we immediately fill up the pipe again plus 1 extra byte, 
    # i.e. 4097 bytes for the default queue size, then NPFS will 
    # internally queue a pending IRP. We're synchronous, so the 
    # I/O manager will wait for I/O completion. Meanwhile the child
    # has requested to close its end of the pipe. In this case,
    # NPFS will complete the pending IRP with STATUS_PIPE_BROKEN,
    # which maps to WinAPI ERROR_PIPE_BROKEN and C errno EPIPE.
    #
    # On the other hand, if we wait to give the child's close request
    # time to complete, then NPFS will fail our 4097 byte write 
    # immediately with STATUS_PIPE_CLOSING, which maps to WinAPI
    # ERROR_NO_DATA and C errno EINVAL.

    time.sleep(0.0) # STATUS_PIPE_BROKEN / ERROR_PIPE_BROKEN / EPIPE
    #time.sleep(0.5) # STATUS_PIPE_CLOSING / ERROR_NO_DATA / EINVAL

    try:
        p.stdin.write(b'\n' * 4097)
    except OSError:
        ntstatus = RtlGetLastNtStatus()
        if ntstatus == 0xC000_00B1:
           print('NT Status: STATUS_PIPE_CLOSING\n')
        elif ntstatus == 0xC000_014B:
            print('NT Status: STATUS_PIPE_BROKEN\n')
        else:
            print('NT Status: {}\n'.format(ntstatus, '#010x'))
        raise

This could be addressed by improving our exception handling to look at the C runtime's _doserrno [1] value for EINVAL errors, in order to map ERROR_NO_DATA to EPIPE instead of EINVAL. Only two NT status codes are mapped to ERROR_NO_DATA, and both are pipe related (STATUS_PIPE_CLOSING and STATUS_PIPE_EMPTY), so using EPIPE should be fine.

[1]: https://docs.microsoft.com/en-us/cpp/c-runtime-library/errno-doserrno-sys-errlist-and-sys-nerr?view=vs-2017

It's still the case that we have to guess that a generic EINVAL (22) is actually EPIPE. Low-level wrapper functions in Python/fileutils.c, such as _Py_read() and _Py_write_impl(), should try to get a better error code by calling Python's custom winerror_to_errno() mapping function on the CRT's _doserrno value. For example:

_Py_read:

    #ifdef MS_WINDOWS
            n = read(fd, buf, (int)count);
            if (n < 0 && errno == EINVAL) {
                errno = winerror_to_errno(_doserrno);
            }

_Py_write_impl:

    #ifdef MS_WINDOWS
                n = write(fd, buf, (int)count);
                if (n < 0 && errno == EINVAL) {
                    errno = winerror_to_errno(_doserrno);
                }

This maps ERROR_NO_DATA to EPIPE.