classification
Title: tempfile.NamedTemporaryFile can close the file too early, if not assigning to a variable
Type: behavior Stage: resolved
Components: IO, Library (Lib) Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Arfrever, Zdeněk.Pavlas, ctheune, georg.brandl, haypo, jort.bloem, jstasiak, martin.panter, meador.inge, mmarkk, ncoghlan, pitrou, python-dev, r.david.murray, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-08-29 20:25 by jort.bloem, last changed 2014-05-26 06:27 by mmarkk. This issue is now closed.

Files
File name Uploaded Description Edit
tempfile.patch jstasiak, 2013-09-05 08:41 review
tempfile_simplify.patch serhiy.storchaka, 2013-09-05 12:03 review
tempfile_wrap_methods.patch serhiy.storchaka, 2013-12-15 21:09 review
tempfile_lifetime.patch pitrou, 2013-12-21 20:41
Messages (22)
msg196485 - (view) Author: jort bloem (jort.bloem) Date: 2013-08-29 20:25
Issue occurs in the following code: tempfile.NamedTemporaryFile(dir=".",delete=False).write("hello")

The NamedTemporaryFile is deleted before the write is complete; deleting the object closes the file, so the write fails.
msg196496 - (view) Author: jort bloem (jort.bloem) Date: 2013-08-29 22:36
Sorry, my python-foo is not quite up to providing a solution. A partial solution would be for the file not to be closed at NamedTemporaryFile destruction, if delete==False. If delete==True, then there is no point writing to a file (though it shouldn't raise an exception, either).

Alternatively, on a unix based system, the file could be deleted, but left open. Read & write could continue as per normal, and the file would be closed automatically later.
msg196501 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-08-29 23:33
The method call should keep the file object alive until it completes (due to the self reference). What behaviour prompted the conclusion that it is being closed early?

If the symptom is that the data isn't being written to disk, we may have a missing flush() call somewhere causing a race condition.
msg196504 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-08-29 23:56
Confirmed in trunk:

>>> tempfile.NamedTemporaryFile(dir=".",delete=False).write(b"hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: write to closed file
msg196507 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-08-30 03:21
'tempfile.NamedTemporaryFile' returns an instance of '_TemporaryFileWrapper' that wraps the created file object.  The wrapper object implements '__getattr__' and we end up with a situation like the following simplified example where the wrapper gets destroyed before the looked up method is called:

$ cat test.py 
class S(object):
   def __init__(self):
      print("S.__init__")   def __del__(self):      print("S.__del__")
   def foo(self):
      print("S.foo")

class T(object):
   def __init__(self, s):
      self.s = s
      print("T.__init__")
   def __getattr__(self, name):
        s = self.__dict__['s']
        a = getattr(s, name)
        print("__getattr__('%s')" % name)
        if not isinstance(a, int):
            setattr(self, name, a)
        return a
   def __del__(self):
      print("T.__del__")
T(S()).foo()
$ ./python test.py 
S.__init__
T.__init__
__getattr__('foo')
T.__del__
S.foo
S.__del__
msg196952 - (view) Author: jort bloem (jort.bloem) Date: 2013-09-04 20:46
I am only new to Python, but...

Having looked at the code, I am surprised that the _TemporaryFileWrapper is not a subclass of "file". Surely that would be the "python" way, and would solve this problem, too? __getattr__ would no longer be needed. The opening of the file is within the library, so _os.open() could be replaced with file()... It would require rewriting/reorganising chunks of this code...
msg196985 - (view) Author: Jakub Stasiak (jstasiak) * Date: 2013-09-05 08:41
I don't see an obvious way of solving that. One thing I could think of is creating a wrapper for file method being returned from __getattr__ that holds reference to _TemporaryFileWrapper instance until the method gets called, please find a patch attached.
msg196991 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-05 12:03
Yet one way is just remove __del__() and monkey-patch underlied file.
msg196992 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-09-05 12:18
I was experimenting with something similar to what Jakub has.  Rebinding the call the the wrapper seems like a simple and clean way to do it.  Although, I wasn't absolutely sure whether this approach covers all cases.
msg197016 - (view) Author: Meador Inge (meador.inge) * (Python committer) Date: 2013-09-05 18:34
The monkey-patched version breaks the auto-close on __del__ feature:

[meadori@li589-207 cpython]$ ./python 
Python 3.4.0a1+ (default:c41c68a18bb6, Sep  5 2013, 17:51:03) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> tempfile.NamedTemporaryFile(dir=".",delete=False).write(b"hello")
__main__:1: ResourceWarning: unclosed file <_io.BufferedRandom name=4>
5

Also run the test suite with '-R :' and you will see tons of resource warnings.

Breaking auto-close is because now unlinking *and* closing are being guarded by 'delete'.  Even if that is fixed you still run into resource leak problems.  Looks like the monkey-patching introduces a reference cycle.

The wrapper binding approach works fine in all the tests I did.
msg197019 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-09-05 19:58
> The monkey-patched version breaks the auto-close on __del__ feature:

Files were closed. Just with a warning. I consider this as an enhancement. If you don't close file explicitly you will get a leak on non-refcounted Python implementations. It's true for both ordinal files so tempfiles. Perhaps we should add a warning in any __del__ method (even if we will commit Jakub's patch, this way LGTM too).
msg204723 - (view) Author: Zdeněk Pavlas (Zdeněk.Pavlas) Date: 2013-11-29 15:02
Hit this issue too, but with the "read" method.  I think that providing a custom read() and write() methods in the wrapper class that override __getattr__ is a sufficient solution. These are the attributes most likely to be used as the "tail".
msg206257 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-15 20:58
Jakub's patch has a bug:

>>> import tempfile
>>> f = tempfile.NamedTemporaryFile(dir=".",delete=False)
>>> write = f.write
>>> write
<function BufferedRandom.write at 0xb716b4f4>
>>> write2 = f.write
>>> write2
<built-in method write of _io.BufferedRandom object at 0xb721deac>
>>> del f
>>> write(b'foo')
3
>>> del write
>>> write2(b'bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: write to closed file
msg206258 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-15 21:09
Here is fixed Jakub's patch.
msg206554 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-18 21:37
Unfortunately last patch causes resource leak on 3.3+.

$ ./python -Wall -m test.regrtest test_tempfile
[1/1] test_tempfile
1 test OK.
sys:1: ResourceWarning: gc: 5 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them
msg206776 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-21 20:33
Here is a proper patch (for 3.3). Untested under Windows, but should work.
msg206778 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-21 20:42
Oops, removed two debug lines.
msg206780 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-12-21 21:04
LGTM. Thank you Antoine.
msg206781 - (view) Author: Roundup Robot (python-dev) Date: 2013-12-21 21:19
New changeset f3b7a76fb778 by Antoine Pitrou in branch '3.3':
Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called.
http://hg.python.org/cpython/rev/f3b7a76fb778

New changeset d68ab2eb7a77 by Antoine Pitrou in branch 'default':
Issue #18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called.
http://hg.python.org/cpython/rev/d68ab2eb7a77
msg206782 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-12-21 21:21
Fixed in 3.3 and 3.4. I think 2.7 is irrelevant at this point.
msg216506 - (view) Author: Christian Theune (ctheune) * Date: 2014-04-16 17:26
#15002 uses this patch to fix a similar wrapping problem in urllib.

Also, this affects 2.7 as well and #15002 does report the problem for 2.7. I'd like to get this fix backported. Would that be OK?
msg219138 - (view) Author: Марк Коренберг (mmarkk) * Date: 2014-05-26 06:27
Is issue 21579 fixed in that bug? too many letters..sorry...
History
Date User Action Args
2014-05-26 06:27:57mmarkksetnosy: + mmarkk
messages: + msg219138
2014-04-17 05:52:33Arfreversettitle: tempfile.NamedTemporaryFile can close the file too early, if not assigning to a variablwe -> tempfile.NamedTemporaryFile can close the file too early, if not assigning to a variable
2014-04-16 17:26:01ctheunesetversions: + Python 2.7, Python 3.5, - Python 3.3
nosy: + ctheune
title: tempfile.NamedTemporaryFile can close the file too early, if not assigning to a variable -> tempfile.NamedTemporaryFile can close the file too early, if not assigning to a variablwe
messages: + msg216506

2013-12-21 21:21:19pitrousetstage: commit review -> resolved
versions: - Python 2.7
2013-12-21 21:21:12pitrousetstatus: open -> closed
resolution: fixed
messages: + msg206782
2013-12-21 21:19:20python-devsetnosy: + python-dev
messages: + msg206781
2013-12-21 21:04:12serhiy.storchakasetassignee: serhiy.storchaka -> pitrou
messages: + msg206780
stage: patch review -> commit review
2013-12-21 20:42:01pitrousetmessages: + msg206778
2013-12-21 20:41:38pitrousetfiles: - tempfile_lifetime.patch
2013-12-21 20:41:32pitrousetfiles: + tempfile_lifetime.patch
2013-12-21 20:33:59pitrousetfiles: + tempfile_lifetime.patch
nosy: + pitrou
messages: + msg206776

2013-12-18 21:37:39serhiy.storchakasetmessages: + msg206554
2013-12-15 21:09:47serhiy.storchakasetfiles: + tempfile_wrap_methods.patch
assignee: serhiy.storchaka
messages: + msg206258
2013-12-15 20:58:37serhiy.storchakasetmessages: + msg206257
2013-11-30 06:07:37martin.pantersetnosy: + martin.panter
2013-11-29 21:04:25Arfreversetnosy: + Arfrever
2013-11-29 15:02:41Zdeněk.Pavlassetnosy: + Zdeněk.Pavlas
messages: + msg204723
2013-09-05 19:58:46serhiy.storchakasetmessages: + msg197019
2013-09-05 18:34:42meador.ingesetmessages: + msg197016
2013-09-05 12:18:31meador.ingesetmessages: + msg196992
2013-09-05 12:03:41serhiy.storchakasetstage: patch review
2013-09-05 12:03:24serhiy.storchakasetfiles: + tempfile_simplify.patch

messages: + msg196991
2013-09-05 08:41:53jstasiaksetfiles: + tempfile.patch

nosy: + jstasiak
messages: + msg196985

keywords: + patch
2013-09-04 20:46:24jort.bloemsetmessages: + msg196952
2013-08-30 03:21:58meador.ingesetmessages: + msg196507
2013-08-30 02:48:47r.david.murraysetnosy: + r.david.murray
2013-08-29 23:56:42meador.ingesetnosy: + meador.inge
messages: + msg196504
2013-08-29 23:33:40ncoghlansetmessages: + msg196501
2013-08-29 22:36:18jort.bloemsetmessages: + msg196496
2013-08-29 20:56:00serhiy.storchakasetnosy: + georg.brandl, ncoghlan, serhiy.storchaka

components: + Library (Lib), IO
versions: + Python 3.3, Python 3.4, - Python 2.6
2013-08-29 20:27:02hayposetnosy: + haypo
2013-08-29 20:25:43jort.bloemcreate