Title: asyncio.readline - add customizable line separator
Type: enhancement Stage:
Components: asyncio Versions: Python 3.6, Python 3.4, Python 3.5
Status: closed Resolution: duplicate
Dependencies: Superseder: Add new StreamReader.readuntil() method
View: 26050
Assigned To: Nosy List: gvanrossum, martin.panter, socketpair, yselivanov
Priority: normal Keywords:

Created on 2015-11-27 21:26 by socketpair, last changed 2016-01-13 17:01 by vstinner. This issue is now closed.

Messages (8)
msg255488 - (view) Author: Марк Коренберг (socketpair) * Date: 2015-11-27 21:26
In Python3.5 asyncio.StreamReader.readline() can separate lines only by b'\n'. For some task, I want actually "read until separator", where separator is another byte.

Since I cannot create pull-request, I will write just here what I want:

def readline(self, separator=b'\n'):
    if not isinstance(separator, bytes) or len(separator) != 1:
        raise ValueError(separator)
    ichar = self._buffer.find(separator)
msg255500 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-11-27 22:49
I like the idea. It's common in protocols to have different kinds of
"terminators" to mark the end of a message. C strings use a null byte
for example. I would be trivial to accept other terminators, since we
don't store received bytes as lines, but as a raw byte string
(technically in a bytearray).

> if not isinstance(separator, bytes) or len(separator) != 1:

Yes, we should only reject empty terminator string. But terminator
strings of more than 1 character are fine, no? I don't think that we
need to put an arbitrary limitation here.

> Since I cannot create pull-request, I will write just here what I want:

Oh, what is your problem? You can use
msg255577 - (view) Author: Марк Коренберг (socketpair) * Date: 2015-11-29 15:37
When I reported that bug, I want to suggest multibyte separators, but implementation is much complex than current one. For example, we should decide what to do if partial separator is read and EOF occur.

So, I think we should merge ability to use custom one-byte separator, and next create another pull-request to make support of multibyte separator (like '\r\n').

Okay, I can make pull request on asyncio, but asyncio now is the part of Python. Does not it ?
msg255584 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2015-11-29 18:21
If you hit EOF in the middle of a partial separator, treat it as a premature EOF just as if you hit EOF before the separator -- the record is not properly terminated so it shouldn't be accepted.

I prefer a single PR that has the full code.

We keep the CPython Lib version and the asyncio repo synchronized, a PR for the asyncio repo is easier to handle for me.
msg255587 - (view) Author: Марк Коренберг (socketpair) * Date: 2015-11-29 20:01
Well, is ready :)
msg256766 - (view) Author: Марк Коренберг (socketpair) * Date: 2015-12-20 15:49
ReadStream.read_until() has been implemented instead enhancing readline().
msg258129 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2016-01-13 03:45
The same Git Hub pull request is mentioned in Issue 26050, so I am guessing this is now a duplicate.
msg258157 - (view) Author: Марк Коренберг (socketpair) * Date: 2016-01-13 16:52
During development, we decide not to change readline() function. So, yes, this issue is closed.
Date User Action Args
2016-01-13 17:01:32vstinnersetnosy: - vstinner
2016-01-13 16:52:12socketpairsetmessages: + msg258157
2016-01-13 03:45:42martin.pantersetstatus: open -> closed

nosy: + martin.panter
messages: + msg258129

superseder: Add new StreamReader.readuntil() method
resolution: wont fix -> duplicate
2015-12-20 15:49:05socketpairsetresolution: wont fix
messages: + msg256766
2015-11-29 20:01:06socketpairsetmessages: + msg255587
2015-11-29 18:21:52gvanrossumsetmessages: + msg255584
2015-11-29 15:37:29socketpairsetmessages: + msg255577
2015-11-27 22:49:32vstinnersetmessages: + msg255500
2015-11-27 21:26:38socketpaircreate