classification
Title: asyncio.StreamReader hangs when reading from pipe and other process exits unexpectedly
Type: Stage:
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, jaswdr, kormang, yselivanov
Priority: normal Keywords:

Created on 2021-04-11 12:17 by kormang, last changed 2021-05-07 09:54 by kormang.

Messages (3)
msg390778 - (view) Author: Marko (kormang) Date: 2021-04-11 12:17
When using asyncio to read from pipe, if process on the other side of pipe crashes read operation hangs indefinitely.

Example:


import asyncio
import contextlib
import os
import signal
import time

prx, ctx = os.pipe()

read_transport = None
read_stream = None

async def _connect_read(loop):
    global read_transport
    global read_stream
    stream_reader = asyncio.StreamReader()
    def protocol_factory():
        return asyncio.StreamReaderProtocol(stream_reader)
    rpipe = os.fdopen(prx, mode='r')

    transport, _ = await loop.connect_read_pipe(protocol_factory, rpipe)
    read_transport = transport
    read_stream = stream_reader

def close():
    read_transport.close()

@contextlib.asynccontextmanager
async def connect():
    try:
        loop = asyncio.get_event_loop()
        await _connect_read(loop)
        yield
    finally:
        close()

cpid = os.fork()
if cpid == 0:
    os.kill(os.getpid(), signal.SIGKILL)
    os.write(ctx, b'A')
    time.sleep(10.0)
else:
    async def read_from_child():
        async with connect():
            input = await read_stream.read(1)
            print('Parent received: ', input)

    asyncio.run(read_from_child())
msg392830 - (view) Author: Jonathan Schweder (jaswdr) * Date: 2021-05-03 18:35
@kormang this is an expected behaviour, this is a problem even for the OS level, just because it is impossible to know when the reader needs to stop waiting, the best option here is to implement some timeout mechanism.
msg393180 - (view) Author: Marko (kormang) Date: 2021-05-07 09:54
@jaswdr Hm, this is was somewhat unexpected for me, since when reading directly from pipe, and EOF is sent.

Timeout was somewhat awkward in my case, since I don't know when other process will start sending, and how long it would take. On the other hand, I use asyncio loop, and can do this asynchronously, so I get notified when child process dies, by other means, and close the stream. So there are plenty of possible workarounds, but I'm not sure it is impossible to solve the problem on the library level yet. It would take more digging into implementation from my side, however.
History
Date User Action Args
2021-05-07 09:54:28kormangsetmessages: + msg393180
2021-05-03 18:35:29jaswdrsetnosy: + jaswdr
messages: + msg392830
2021-04-16 20:23:14terry.reedysetnosy: + asvetlov, yselivanov
2021-04-11 12:17:49kormangcreate