classification
Title: Can't read data from Transport after asyncio.SubprocessStreamProtocol closes
Type: behavior Stage: resolved
Components: asyncio Versions:
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: SethMichaelLarson, yselivanov
Priority: normal Keywords:

Created on 2017-03-02 22:24 by SethMichaelLarson, last changed 2017-03-31 16:36 by dstufft. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 405 merged SethMichaelLarson, 2017-03-02 22:24
PR 415 closed yselivanov, 2017-03-03 04:33
PR 416 merged yselivanov, 2017-03-03 04:34
PR 703 larry, 2017-03-17 21:00
PR 552 closed dstufft, 2017-03-31 16:36
Messages (4)
msg288839 - (view) Author: Seth Michael Larson (SethMichaelLarson) * Date: 2017-03-02 22:24
Copied from https://github.com/python/asyncio/issues/484

"""
From https://bugs.python.org/issue23242#msg284930

The following script is used to reproduce the bug:

import asyncio

async def execute():
    process = await asyncio.create_subprocess_exec(
        "timeout", "0.1", "cat", "/dev/urandom", stdout=asyncio.subprocess.PIPE)

    while True:
        data = await process.stdout.read(65536)
        print('read %d bytes' % len(data))
        if data:
            await asyncio.sleep(0.3)
        else:
            break

asyncio.get_event_loop().run_until_complete(execute())

will produce following output and terminate with exception:

read 65536 bytes
read 65536 bytes
Traceback (most recent call last):
  File "read_subprocess.py", line 18, in <module>
    asyncio.get_event_loop().run_until_complete(execute())
  File "/usr/lib/python3.6/asyncio/base_events.py", line 466, in run_until_complete
    return future.result()
  File "read_subprocess.py", line 9, in execute
    data = await process.stdout.read(65536)
  File "/usr/lib/python3.6/asyncio/streams.py", line 634, in read
    self._maybe_resume_transport()
  File "/usr/lib/python3.6/asyncio/streams.py", line 402, in _maybe_resume_transport
    self._transport.resume_reading()
  File "/usr/lib/python3.6/asyncio/unix_events.py", line 401, in resume_reading
    self._loop._add_reader(self._fileno, self._read_ready)
AttributeError: 'NoneType' object has no attribute '_add_reader'

When the process exits https://github.com/python/asyncio/blob/master/asyncio/unix_events.py#L444 is called which sets this._loop = None
Next time read() is called on the pipe the above exception is thrown.
I have tried to fix this issue myself but would sometimes have read terminate too early and miss the last chunks of data.
"""

- BotoX
msg290327 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-24 23:04
New changeset f7f024a721d53978d03129e8eb5111d4f74534a9 by Yury Selivanov (Seth M. Larson) in branch '3.5':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
https://github.com/python/cpython/commit/f7f024a721d53978d03129e8eb5111d4f74534a9
msg290333 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-24 23:07
New changeset 604faba1db724951ee440337099d111e3ecade49 by Yury Selivanov (Seth M. Larson) in branch '3.6':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
https://github.com/python/cpython/commit/604faba1db724951ee440337099d111e3ecade49
msg290335 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2017-03-24 23:08
New changeset 481cb70a724687d12553d38a749c16034af68a1e by Yury Selivanov (Seth M. Larson) in branch 'master':
bpo-29704: Fix asyncio.SubprocessStreamProtocol closing (#405)
https://github.com/python/cpython/commit/481cb70a724687d12553d38a749c16034af68a1e
History
Date User Action Args
2017-03-31 16:36:23dstufftsetpull_requests: + pull_request968
2017-03-24 23:08:27yselivanovsetmessages: + msg290335
2017-03-24 23:07:23yselivanovsetmessages: + msg290333
2017-03-24 23:04:15yselivanovsetmessages: + msg290327
2017-03-17 21:00:33larrysetpull_requests: + pull_request592
2017-03-03 23:12:44yselivanovsetstatus: open -> closed
resolution: fixed
stage: resolved
2017-03-03 04:34:25yselivanovsetpull_requests: + pull_request347
2017-03-03 04:33:41yselivanovsetpull_requests: + pull_request346
2017-03-02 22:33:08gvanrossumsetnosy: - gvanrossum
2017-03-02 22:28:41SethMichaelLarsonsetnosy: + gvanrossum
type: behavior
components: + asyncio
2017-03-02 22:24:23SethMichaelLarsoncreate