classification
Title: io's close() not handling errors correctly
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.3, Python 3.2, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, eckhardt, haypo, pitrou
Priority: critical Keywords: patch

Created on 2009-01-05 10:29 by eckhardt, last changed 2011-08-10 20:05 by meador.inge.

Files
File name Uploaded Description Edit
python-2.7-fileio-close-test.0.patch eckhardt, 2009-01-05 10:51 patch review
python-2.7-fileio-close-fix.0.patch eckhardt, 2009-01-05 14:03 patch review
fileio_close.patch haypo, 2011-06-20 14:42 patch to python 3.3 review
Messages (12)
msg79120 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2009-01-05 10:29
In _fileio.c, there is the following comment: "Returns 0 on success,
errno (which is < 0) on failure." The problem here is the claim that
errno ever was less than zero, which is simply wrong.

You can see this being a problem with the following few lines:

import os, io
fd = os.open( "some existing file", os.O_RDONLY)
s1 = os.fdopen(fd)
s2 = io.open(fd)
os.close(fd)
s1.close()
s2.close()

The call to close() pulls the file from under the feet of the two stream
objects, but only the one opened with os.fdopen() actually detects that.
For the second one, errno is set, but to a positive value which isn't
detected correctly.
msg79125 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2009-01-05 10:51
This adds a test that the close() correctly fails with an IOError.
msg79139 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-01-05 13:05
Strangely enough, it works in py3k.
msg79141 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2009-01-05 13:42
In Python 3.x, internal_close() returns -1 on error (if close() 
returns a negative value) and 0 on success.

In Python 2.x, internal_closes() returns the errno value of close() on 
error, or 0 on success. fileio_close() tests if the error value is 
negative instead of if it's not null or at least it's strict positive.

Python 3.x code looks better. I tried to backport the internal_close() 
changes but I get a segfault on PyErr_WriteUnraisable() :-p (close() 
error in fileio_dealloc)
msg79149 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2009-01-05 14:03
I have a patch here which passes the proposed change to the tests in
trunk. A few notes up front though, since the patch isn't good yet:

1. The handling of the error in internal_close() isn't really good. What
I'm also not sure about is whether it is correct to set the error there,
the old code sometimes did set an error as reaction to internal_close()
failing and sometimes not.
2. There is one place in dircheck() where I think that the error should
be ignored, but I'm not sure about that.
3. This patch fixes another issue, and that is that internal_close()
would call close() even if 'closefd' was not set. This e.g. happens when
__init__ is called explicitly. This case is missing a test though.
4. I wonder if internal_close() should reset only the 'fd' field or if
it should also reset 'readable', 'writable', 'seekable', too, i.e. apart
from 'weakreflist' do the same as fileio_new() does.
msg79328 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2009-01-07 12:38
> 1. The handling of the error in internal_close() isn't really good. What
> I'm also not sure about is whether it is correct to set the error there,
> the old code sometimes did set an error as reaction to internal_close()
> failing and sometimes not.

internal_close() is used in:
 - fileio_close(): raise an IOError(errno) on internal_close() failure
 - dircheck(): ignore internal_close() failure, whereas Python 3.x raise 
   a IOError(errno) on internal_close() failure
 - file_init(): catch internal_close() failure but don't set an exception,
   whereas Python 3.x set a IOError(errno)
 - fileio_dealloc(): PySys_WriteStderr(...) in Python 2.x,
   PyErr_WriteUnraisable(self) in Python 3.x

But Python 2.x all errors are ignored because the test (errno < 0) is wrong...

I always prefer Python 3.x behaviour: internal_close() is responsible to raise 
an exception.

> 2. There is one place in dircheck() where I think that the error should
> be ignored, but I'm not sure about that.

I rarely a good idea to ignore errors, I prefer to raise a different error if 
the file can not be closed.

> 3. This patch fixes another issue, and that is that internal_close()
> would call close() even if 'closefd' was not set. This e.g. happens when
> __init__ is called explicitly. This case is missing a test though.

Python 3.x is also affected by this issue: file_init() and dircheck() ignore 
self->closefd.

passwd = open("/etc/passwd")
f = _FileIO(fd, closefd=False)
f.__init__(fd, closefd=False) => close passwd file!

> 4. I wonder if internal_close() should reset only the 'fd' field or if
> it should also reset 'readable', 'writable', 'seekable', too, i.e. apart
> from 'weakreflist' do the same as fileio_new() does.

seekable(), readable() and writable() raise an error if the file is closed. I 
think it's fine to leave the internal attributes unchanged.
msg79331 - (view) Author: Ulrich Eckhardt (eckhardt) Date: 2009-01-07 13:11
>> 4. I wonder if internal_close() should reset only the 'fd' field
>> or if it should also reset 'readable', 'writable', 'seekable'
>
> seekable(), readable() and writable() raise an error if the file
> is closed. I think it's fine to leave the internal attributes
> unchanged.

That's not what I'm concerned about. What I'm concerned about is this:

f = _FileIO(open("some text file"))
print f.seekable()
f.__init__(open("some pipe"))
print f.seekable()

The cached value whether the file was seekable survives the __init__ 
call but is actually wrong then. Of course, one could also reset the 
value in __init__. Whether the file is readable or writable is 
different, since those values are supplied via the mode string and the 
code by the user and not determined automatically.


I think that I just found another buglet, too: when supplying a file 
descriptor, dircheck() isn't actually called. os.open() under 2.5 on 
Debian/Linux at least allows opening '.', trunk under MS Windows fails 
with "permission denied".
msg110204 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-13 15:25
Please note the comment from Victor:-
"Python 3.x is also affected by this issue: file_init() and dircheck() ignore self->closefd."
msg111994 - (view) Author: Mark Lawrence (BreamoreBoy) Date: 2010-07-29 20:31
Might as well close as nobody appears interested.
msg138724 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-06-20 14:42
fileio_close.patch (for Python 3.3): Fix FileIO.__init__() to not close the file if closefd=False and the constructor is called twice (or more).
msg138725 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-06-20 14:44
fileio_close.patch should maybe use os.open() (to create the fd) and os.fstat() (to check that the fd is not closed).
msg138727 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2011-06-20 15:12
Status of this issue:
 - io.FileIO.close() raises IOError with Python 2.7, 3.1, 3.2 and 3.3 (e.g. if the underlying file descriptor has been closed), it doesn't with Python 2.6
 - If FileIO constructor is called twice, the file is closed at the second call even if closefd was False at the first call: bug in Python 2.6-3.3: fileio_close.patch fixes this bug

So except fixing FileIO constructor (using fileio_close.patch) in Python 2.7, 3.2 and 3.3, there is nothing more to do.
History
Date User Action Args
2011-08-10 20:05:23meador.ingesetstage: needs patch -> patch review
2011-06-20 15:12:41hayposetmessages: + msg138727
versions: + Python 3.3
2011-06-20 14:44:19hayposetmessages: + msg138725
2011-06-20 14:42:00hayposetfiles: + fileio_close.patch

messages: + msg138724
2011-06-12 18:31:00terry.reedysetversions: - Python 3.1
2010-07-30 13:24:18brian.curtinsetstatus: pending -> open
2010-07-29 20:31:37BreamoreBoysetstatus: open -> pending

messages: + msg111994
2010-07-13 15:25:32BreamoreBoysetnosy: + BreamoreBoy

messages: + msg110204
versions: + Python 3.1, Python 3.2, - Python 2.6
2009-01-07 13:11:07eckhardtsetmessages: + msg79331
2009-01-07 12:38:52hayposetmessages: + msg79328
2009-01-05 14:03:07eckhardtsetfiles: + python-2.7-fileio-close-fix.0.patch
messages: + msg79149
2009-01-05 13:42:33hayposetmessages: + msg79141
2009-01-05 13:41:19hayposetmessages: - msg79140
2009-01-05 13:38:29hayposetnosy: + haypo
messages: + msg79140
2009-01-05 13:05:00pitrousetpriority: critical
nosy: + pitrou
messages: + msg79139
stage: needs patch
2009-01-05 10:51:16eckhardtsetfiles: + python-2.7-fileio-close-test.0.patch
keywords: + patch
messages: + msg79125
2009-01-05 10:29:49eckhardtcreate