classification
Title: asyncio Stream Reader Blocks on read when data fetched is less than limit
Type: behavior Stage: resolved
Components: asyncio Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: asvetlov, skorpeo, yselivanov
Priority: normal Keywords:

Created on 2018-05-27 22:34 by skorpeo, last changed 2018-05-27 23:39 by yselivanov. This issue is now closed.

Files
File name Uploaded Description Edit
pty_test.py skorpeo, 2018-05-27 22:34 demonstraton of blocking behaviour
Messages (6)
msg317810 - (view) Author: (skorpeo) Date: 2018-05-27 22:34
I humbly submit what I think may be a bug in the asyncio.StreamReader.read() function.  When n=-1 is supplied and the data is less than self._limit the read function creates a future and never wakes it up.  I believe the culprit is https://github.com/python/cpython/blob/9d3627e311211a1b4abcda29c36fe4afe2c46532/Lib/asyncio/streams.py#L632.  To fix the issue a condition is added to break out of the loop if the data read is less than the limit.  I can only attach one file so I am providing the fix here for asyncio streams.py:  

blocks.append(block)           # existing
if len(block) < self._limit:   # new
    break                      # new

I have also attached a test file that shows the blocking behavior which is alleviated with the above fix.  Finally, I am not sure how to handle a situation where the data is exactly equal to the limit and no subsequent data is sent.
msg317811 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-27 22:45
"if not block:" means EOF and replacing it with "if len(block) < self._limit:" would break everything.

There might be another bug here (with TTY) or, maybe, there's a bug in pty_test.py.  For example, I don't see how you closing reader_pipe or writer_pipe.
msg317812 - (view) Author: (skorpeo) Date: 2018-05-27 23:02
Yes makes sense, it could be another bug with TTY or just my error and I
did preface that I am submitting this humbly.  You are also correct that
there is no clean up for closing the pipes.

On Mon, May 28, 2018 at 1:45 AM, Yury Selivanov <report@bugs.python.org>
wrote:

>
> Yury Selivanov <yselivanov@gmail.com> added the comment:
>
> "if not block:" means EOF and replacing it with "if len(block) <
> self._limit:" would break everything.
>
> There might be another bug here (with TTY) or, maybe, there's a bug in
> pty_test.py.  For example, I don't see how you closing reader_pipe or
> writer_pipe.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33662>
> _______________________________________
>
msg317814 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-27 23:03
> You are also correct that there is no clean up for closing the pipes.

It's not just about the cleanup. If you don't close the pipes, they will be open forever, so there will be no EOF for which read(-1) will wait forever.
msg317815 - (view) Author: (skorpeo) Date: 2018-05-27 23:37
yes, in this case they were meant to stay open to write and read multiple
messages.  I was hoping to read data when it is available, the other work
around was to specify n, but that also blocked once there was no more data
to be fetched....Either way I will stick to queues and this appears to be
expected behavior so you can disregard the report.  I should have known
better that I wasn't going to find a bug in python :)

On Mon, May 28, 2018 at 2:03 AM, Yury Selivanov <report@bugs.python.org>
wrote:

>
> Yury Selivanov <yselivanov@gmail.com> added the comment:
>
> > You are also correct that there is no clean up for closing the pipes.
>
> It's not just about the cleanup. If you don't close the pipes, they will
> be open forever, so there will be no EOF for which read(-1) will wait
> forever.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue33662>
> _______________________________________
>
msg317816 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2018-05-27 23:39
> I should have known better that I wasn't going to find a bug in python :)

We have a lot of bugs, as any other software :)  Don't be afraid to open issues.

I'll close this bug, feel free to re-open.
History
Date User Action Args
2018-05-27 23:39:51yselivanovsetstatus: open -> closed
resolution: not a bug
messages: + msg317816

stage: resolved
2018-05-27 23:37:07skorpeosetmessages: + msg317815
2018-05-27 23:03:56yselivanovsetmessages: + msg317814
2018-05-27 23:02:37skorpeosetmessages: + msg317812
2018-05-27 22:45:56yselivanovsetmessages: + msg317811
2018-05-27 22:34:41skorpeocreate