classification
Title: Discrepancy between io.StringIO and _pyio.StringIO with univeral newlines
Type: behavior Stage: resolved
Components: IO Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, hynek, pitrou, python-dev, serhiy.storchaka, stutzbach
Priority: normal Keywords: patch

Created on 2014-01-29 09:16 by serhiy.storchaka, last changed 2014-02-03 06:39 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
stringio_getvalue.patch pitrou, 2014-02-02 21:39
stringio_newline.patch serhiy.storchaka, 2014-02-02 22:31 review
Messages (11)
msg209625 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-01-29 09:16
>>> import io, _pyio
>>> io.StringIO('a\nb\r\nc\rd', newline=None)
<_io.StringIO object at 0xb707c734>
>>> io.StringIO('a\nb\r\nc\rd', newline=None).getvalue()
'a\nb\nc\nd'
>>> _pyio.StringIO('a\nb\r\nc\rd', newline=None).getvalue()
'a\nb\r\nc\rd'
>>> s = io.StringIO(newline=None); s.write('a\nb\r\nc\rd'); s.getvalue()
8
'a\nb\nc\nd'
>>> s = _pyio.StringIO(newline=None); s.write('a\nb\r\nc\rd'); s.getvalue()
8
'a\nb\r\nc\rd'
msg210019 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-02 21:39
It's a bug in _pyio.StringIO.getvalue(). Attaching patch.
msg210028 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-02 22:02
I rather think that it's a bug in _io.StringIO.getvalue().
msg210031 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-02 22:06
With the patch, getvalue() is consistent with read() and other methods.
Without the patch, _pyio.StringIO.getvalue() returns a different value from _pyio.StringIO.read() *and* from _io.StringIO.getvalue().
Changing _pyio.StringIO.getvalue() is the path of least resistance here.
msg210033 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-02 22:09
In other words, the bug is that _pyio.StringIO.getvalue() doesn't do any newline conversion; the patch fixes that.
msg210041 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-02 22:31
But how then other values of newline except '\n' can be useful?

The problem is that newline converting is applied twice, in write() and in read(). When constructor uses write() and getvalue() returns same value as read(), we have no chance to get newlines encoded or decoded only once. Current results for newline != '\n' looks meanless to me.

Here is my half-baked patch. It fixes only _pyio.StringIO, but _io.StringIO are still partially broken. The patch also contains new tests.
msg210043 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-02 22:36
> Current results for newline != '\n' looks meanless to me.

They don't look meaningless to me, e.g.:

>>> io.StringIO("a\r\nc\rd", newline=None).getvalue()
'a\nc\nd'
>>> sio = io.StringIO(newline=None); sio.write("a\r\nc\rd"); sio.getvalue()
6
'a\nc\nd'

There may be other cases where they make less sense, but that's a separate issue.
msg210044 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-02 22:40
New changeset 99168e7d4a3d by Antoine Pitrou in branch '3.3':
Issue #20435: Fix _pyio.StringIO.getvalue() to take into account newline translation settings.
http://hg.python.org/cpython/rev/99168e7d4a3d

New changeset aadcc71a4967 by Antoine Pitrou in branch 'default':
Issue #20435: Fix _pyio.StringIO.getvalue() to take into account newline translation settings.
http://hg.python.org/cpython/rev/aadcc71a4967
msg210045 - (view) Author: Roundup Robot (python-dev) Date: 2014-02-02 22:42
New changeset 3e61d8e06ef7 by Antoine Pitrou in branch '2.7':
Issue #20435: Fix _pyio.StringIO.getvalue() to take into account newline translation settings.
http://hg.python.org/cpython/rev/3e61d8e06ef7
msg210047 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-02-02 22:42
I've fixed the reported bug. If other problems need fixing, better open a new issue :-)
msg210079 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-02-03 06:39
Thank you for fixing this bug. Maybe I reported it too hastily.
History
Date User Action Args
2014-02-03 06:39:04serhiy.storchakasetmessages: + msg210079
2014-02-02 22:42:50pitrousetstatus: open -> closed
resolution: fixed
messages: + msg210047

stage: resolved
2014-02-02 22:42:04python-devsetmessages: + msg210045
2014-02-02 22:40:56python-devsetnosy: + python-dev
messages: + msg210044
2014-02-02 22:36:54pitrousetmessages: + msg210043
2014-02-02 22:31:49serhiy.storchakasetfiles: + stringio_newline.patch

messages: + msg210041
2014-02-02 22:09:07pitrousetmessages: + msg210033
2014-02-02 22:06:56pitrousetmessages: + msg210031
2014-02-02 22:02:26serhiy.storchakasetmessages: + msg210028
2014-02-02 21:39:07pitrousetfiles: + stringio_getvalue.patch
keywords: + patch
messages: + msg210019
2014-01-29 09:16:06serhiy.storchakacreate