classification
Title: fileinput, StringIO, and cStringIO do not support the with protocol
Type: behavior Stage: patch review
Components: Extension Modules, Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: Nosy List: Wade.Tattersall, alexandre.vassalotti, belopolsky, brian.curtin, eric.snow, ezio.melotti, georg.brandl, orsenthil, ygale
Priority: low Keywords: easy, patch

Created on 2007-10-16 11:59 by ygale, last changed 2012-11-28 07:00 by eric.snow. This issue is now closed.

Files
File name Uploaded Description Edit
with_StringIO.diff belopolsky, 2008-02-20 17:07 diff against revision 60906
issue1286.patch brian.curtin, 2009-11-17 20:59 patch implementing context manager for fileinput
Messages (18)
msg56497 - (view) Author: Yitz Gale (ygale) Date: 2007-10-16 12:42
The standard idiom for opening a file is now with open...
So I think it should be a goal that this should work with
any built-in file-like object that needs to be closed,
without having to explicitly wrap it in closing().
It certainly should work for fileinput and StringIO - since
these really are files, in some sense.
msg56636 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2007-10-22 01:16
Do you have a use-case for this? In Py3k, I don't think adding support
for the 'with' statement to StringIO makes any sense, since the close()
method does nothing.
msg56643 - (view) Author: Yitz Gale (ygale) Date: 2007-10-22 08:05
These objects are supposed to be drop-in replacements
for file handles. Except in legacy code, the way you
use a file handle is:

with function_to_create_fh as fh:
  <do stuff>

If these objects do not support the with protocol,
the only way to use them is to refactor this code.
That may not even be possible, e.g., if it is in
a library, or it may not be desirable to refactor.

Even if you can refactor it, I don't think you
can call these objects file-like objects if
you can't use them like a file.

Note that in legacy code, you used to write:

fh = function_to_get_fh
try:
  <do stuff>
finally:
  fh.close()

If function_to_get_fh happens to return some other
file-like object and not an actual file,
this still works fine without any refactoring.

You wrote:
> In Py3k, I don't think adding support
> for the 'with' statement to StringIO makes
> any sense, since the close()
> method does nothing.

So do you propse removing the close() method
altogether? Of course the answer is "no",
because then you could not use the object like
a file. The same is true for the with protocol.

(I now retract the words "that needs to be closed"
from my original comment. All file-like objects
should support the with protocol, regardles of
whether their close() method actually does anything.)
msg56678 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2007-10-23 12:31
Makes sense to me.
msg56714 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-10-24 18:30
Let's not get overexcited. I agree that it makes sense for fileinput,
but I disagree about *StringIO; its close() isn't needed to free
resources (since it doesn't use up a scarce resource like a file
descriptor).

Can you whip up a patch for fileinput?  Please include unit tests and
documentation.
msg56740 - (view) Author: Yitz Gale (ygale) Date: 2007-10-25 12:08
I was actually bitten badly by this issue with
StringIO. I added fileinput only as an afterthought.

In an xml.sax app, I needed seek() support for a
codec-wrapped file handle, so I over-wrapped it with
StringIO. The result was that I had to refactor code all over
the place to handle StringIO as a special case. What a
mess!

Why is this getting over-excited? It's a very
lightweight change. You can't beat the cost/benefit ratio.
msg56757 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2007-10-25 22:48
2007/10/25, Yitz Gale <report@bugs.python.org>:
> I was actually bitten badly by this issue with
> StringIO. I added fileinput only as an afterthought.
>
> In an xml.sax app, I needed seek() support for a
> codec-wrapped file handle, so I over-wrapped it with
> StringIO. The result was that I had to refactor code all over
> the place to handle StringIO as a special case. What a
> mess!

I don't understand. What did your code look like after the refactoring?

I find that typically a useful idiom is to have one piece of code
handle opening/closing of streams and let the rest of the code just
deal with streams without ever closing them. E.g.

f = open(filename)
try:
  process(f)
finally:
  f.close()

or, if you want:

with open(filename) as f:
  process(f)

As I don't understand how you are working the StringIO() call into
this I'm still not sure what the issue is.

> Why is this getting over-excited? It's a very
> lightweight change. You can't beat the cost/benefit ratio.

Until you submit a patch it's more work for me. :-)
msg59778 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2008-01-12 02:54
FYI, StringIO and BytesIO, in Python 3K, already support the context
management protocol.
msg62595 - (view) Author: Alexander Belopolsky (belopolsky) * (Python committer) Date: 2008-02-20 17:07
Attached patch implements context management protocol for StringIO.
msg95401 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2009-11-17 20:59
Attached is a patch against trunk r76325 which implements context
manager support for fileinput/FileInput, with tests and doc change.
msg96699 - (view) Author: Senthil Kumaran (orsenthil) * (Python committer) Date: 2009-12-20 17:36
I reviewed the patches attached.
- The patch to add Context Manager support for fileinput.py seems good.
It has docs too.

This discussion did not conclude on the need for Context Manager for
StringIO. With py3k having it, it should be good for py2.7 to provide
the support too. The attached patch seems good enough, Docs can be added
further.
msg96944 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2009-12-28 08:46
FWIW, for the sake of consistency I'm +1 on supporting the context
manager protocol on all file-like objects in the stdlib.
msg112195 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2010-07-31 20:08
For 3.x, the builtin io.StringIO and io.BytesIO already have context manager capability.  Added fileinput in r83359.
msg176329 - (view) Author: Wade Tattersall (Wade.Tattersall) Date: 2012-11-25 04:52
Any chance this patch could be applied to version 2.7? It's still an issue in 2.7.3, even though a suitable patch was supplied 3 years ago.

I understand that it's fixed in python3, but for us poor maintainers of ancient code, it would be convenient to be able to do things like

with StringIO() as test:
    test.write("hi!")
    return test.getvalue()
msg176361 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2012-11-25 14:41
It would be a new feature, and as such forbidden to add in maintenance releases.
msg176512 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-11-28 05:42
Keep in mind that it's pretty easy to roll your own CM wrapper:

@contextlib.contextmanager
def closes(file):
    yield file
    file.close()

Then you can do this:

with closes(StringIO()) as test:
    test.write("hi!")
    return test.getvalue()

This works for 2.5 and up.
msg176513 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-11-28 05:43
http://docs.python.org/2/library/contextlib.html#contextlib.closing
msg176518 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2012-11-28 07:00
borrowed the time machine did we? ;)
History
Date User Action Args
2012-11-28 07:00:51eric.snowsetmessages: + msg176518
2012-11-28 05:43:18ezio.melottisetnosy: + ezio.melotti
messages: + msg176513
2012-11-28 05:42:15eric.snowsetnosy: + eric.snow
messages: + msg176512
2012-11-25 14:41:09georg.brandlsetmessages: + msg176361
2012-11-25 04:52:27Wade.Tattersallsetnosy: + Wade.Tattersall
messages: + msg176329
2010-07-31 20:08:39georg.brandlsetstatus: open -> closed
resolution: accepted
messages: + msg112195
2009-12-28 08:46:30georg.brandlsetmessages: + msg96944
2009-12-20 17:36:47orsenthilsetnosy: + orsenthil
messages: + msg96699
2009-12-04 19:45:14alexandre.vassalottilinkissue7426 superseder
2009-11-17 21:03:15gvanrossumsetnosy: - gvanrossum
2009-11-17 20:59:29brian.curtinsetfiles: + issue1286.patch

nosy: + brian.curtin
messages: + msg95401

keywords: + patch
2009-05-15 01:50:11ajaksu2setstage: patch review
2008-10-04 15:49:31benjamin.petersonsetversions: + Python 2.7, - Python 2.6
2008-02-20 17:07:09belopolskysetfiles: + with_StringIO.diff
nosy: + belopolsky
messages: + msg62595
components: + Library (Lib)
versions: - Python 3.0
2008-01-12 02:54:01alexandre.vassalottisetmessages: + msg59778
2008-01-12 01:27:28akuchlingsetkeywords: + easy
2007-10-25 22:48:59gvanrossumsetmessages: + msg56757
2007-10-25 12:09:00ygalesetmessages: + msg56740
2007-10-24 18:30:38gvanrossumsetpriority: low
nosy: + gvanrossum
messages: + msg56714
versions: - Python 2.5
2007-10-23 12:31:42georg.brandlsetnosy: + georg.brandl
messages: + msg56678
2007-10-22 08:05:51ygalesetmessages: + msg56643
2007-10-22 01:16:28alexandre.vassalottisetnosy: + alexandre.vassalotti
messages: + msg56636
2007-10-16 12:42:36ygalesetmessages: + msg56497
2007-10-16 11:59:57ygalecreate