classification
Title: TextIOWrapper should fall back on read() if read1() doesn't exist
Type: behavior Stage: resolved
Components: Interpreter Core, IO, Library (Lib) Versions: Python 3.2, Python 3.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, anacrolix, eric.araujo, gregory.p.smith, haypo, lukasz.langa, pitrou, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2011-07-20 05:24 by anacrolix, last changed 2011-07-23 20:12 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
textio_rawio.patch pitrou, 2011-07-23 19:03
spnewlines.patch pitrou, 2011-07-23 19:41
Messages (15)
msg140720 - (view) Author: Matt Joiner (anacrolix) Date: 2011-07-20 05:24
>>> a = subprocess.Popen(['cat', '/path/to/text.ini'], stdout=subprocess.PIPE, universal_newlines=True)
>>> b = configparser.ConfigParser()
>>> b.read_file(a.stdout)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/hostname/sig/local/lib/python3.2/configparser.py", line 708, in read_file
    self._read(f, source)
  File "/hostname/sig/local/lib/python3.2/configparser.py", line 994, in _read
    for lineno, line in enumerate(fp, start=1):
AttributeError: '_io.FileIO' object has no attribute 'read1'

Also this fails without universal_readlines, which is not so bad except that the name 'read_file' doesn't exactly indicate this.

I found one mildly related bug: http://bugs.python.org/issue11670
msg140726 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-07-20 12:06
This isn't a problem with ConfigParser, it's a problem with the text file object returned by Subprocess.  You can reproduce the bug by trying to iterate over the TextIOWrapper returned by subprocess.

(It is true, however, that the ConfigParser docs should be specific that it isn't "any file", but "any text file".  If you would like to open a separate issue about that, that would be great).
msg140849 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-22 00:48
The "universal_newlines" feature is rather poorly named in Python 3.x, since it does much more than that (the resulting files yield and expect unicode strings rather than bytes objects).

The problem is that io.TextIOWrapper needs a buffered I/O object, but bufsize is 0 by default in subprocess.Popen(). In the meantime, you can workaround it using a non-zero bufsize in the Popen() constructor. Of course, this has the side-effect of forcing you to call flush() on the stdin stream (if you are using it).
msg140856 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2011-07-22 01:53
So this is a doc bug in subprocess?  Explaining this clearly doesn't sound like much fun...
msg140998 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 17:57
So, as the title indicates, I think we should make TextIOWrapper work with raw IO objects. The reason is so that write() can behave in a totally unbuffered way, which is necessary for Popen to behave appropriately.
msg141000 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 18:16
Hmm, one problem is that the C TextIOWrapper buffers writes. We would need an additional constructor parameter to prevent that :/
msg141002 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 18:33
Another possibility is to provide read1() on RawIO, as a synonym of read().
msg141003 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 19:03
Here is a first patch allowing TextIOWrapper to work with raw IO objects, and adding a "write_through" flag.
msg141005 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-07-23 19:14
"write_through" is not used in _pyio.py, is it expected?
msg141006 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 19:17
> "write_through" is not used in _pyio.py, is it expected?

Yup, because it is actually always write-through (writes are not
cached). The argument is only there so that the signature is the same as
the C version.
msg141008 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-07-23 19:37
Looks good, then.
msg141009 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 19:41
This additional patch improves universal_newlines support in subprocess (still broken with the select- and poll-based loops in communicate()).
msg141011 - (view) Author: Roundup Robot (python-dev) Date: 2011-07-23 19:52
New changeset 9144014028f3 by Antoine Pitrou in branch '3.2':
Issue #12591: Allow io.TextIOWrapper to work with raw IO objects (without
http://hg.python.org/cpython/rev/9144014028f3

New changeset c3b47cdea0d1 by Antoine Pitrou in branch 'default':
Issue #12591: Allow io.TextIOWrapper to work with raw IO objects (without
http://hg.python.org/cpython/rev/c3b47cdea0d1
msg141013 - (view) Author: Roundup Robot (python-dev) Date: 2011-07-23 20:06
New changeset 5cc536fbd7c1 by Antoine Pitrou in branch '3.2':
Issue #12591: Improve support of "universal newlines" in the subprocess
http://hg.python.org/cpython/rev/5cc536fbd7c1

New changeset b616396fa170 by Antoine Pitrou in branch 'default':
Issue #12591: Improve support of "universal newlines" in the subprocess
http://hg.python.org/cpython/rev/b616396fa170
msg141015 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-07-23 20:12
Ok, this should be fixed now. universal newlines support is still broken with communicate(), I've opened issue12623 for that.
History
Date User Action Args
2011-07-23 20:12:23pitrousetstatus: open -> closed
resolution: fixed
messages: + msg141015

stage: needs patch -> resolved
2011-07-23 20:06:08python-devsetmessages: + msg141013
2011-07-23 19:52:29python-devsetnosy: + python-dev
messages: + msg141011
2011-07-23 19:41:36pitrousetfiles: + spnewlines.patch

messages: + msg141009
2011-07-23 19:37:33amaury.forgeotdarcsetmessages: + msg141008
2011-07-23 19:17:12pitrousetmessages: + msg141006
2011-07-23 19:14:17amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg141005
2011-07-23 19:03:25pitrousetfiles: + textio_rawio.patch
keywords: + patch
messages: + msg141003
2011-07-23 18:33:06pitrousetmessages: + msg141002
2011-07-23 18:16:33pitrousetmessages: + msg141000
2011-07-23 17:57:31pitrousetmessages: + msg140998
2011-07-23 17:52:40pitrousettitle: text files returned by subprocess.Popen with universal_newlines=True are not iterable -> TextIOWrapper should fall back on read() if read1() doesn't exist
2011-07-22 01:53:14r.david.murraysetmessages: + msg140856
2011-07-22 00:48:48pitrousetmessages: + msg140849
2011-07-22 00:20:45eric.araujosetnosy: + eric.araujo
2011-07-20 12:06:42r.david.murraysetversions: + Python 3.3
type: behavior

nosy: + pitrou, r.david.murray, lukasz.langa, gregory.p.smith, haypo
title: configparser can't read_file the output of subprocess.Popen -> text files returned by subprocess.Popen with universal_newlines=True are not iterable
messages: + msg140726
stage: needs patch
2011-07-20 05:24:36anacrolixcreate