Issue occurs in the following code: tempfile.NamedTemporaryFile(dir=".",delete=False).write("hello")

The NamedTemporaryFile is deleted before the write is complete; deleting the object closes the file, so the write fails.

Sorry, my python-foo is not quite up to providing a solution. A partial solution would be for the file not to be closed at NamedTemporaryFile destruction, if delete==False. If delete==True, then there is no point writing to a file (though it shouldn't raise an exception, either).

Alternatively, on a unix based system, the file could be deleted, but left open. Read & write could continue as per normal, and the file would be closed automatically later.

The method call should keep the file object alive until it completes (due to the self reference). What behaviour prompted the conclusion that it is being closed early?

If the symptom is that the data isn't being written to disk, we may have a missing flush() call somewhere causing a race condition.

Confirmed in trunk:

>>> tempfile.NamedTemporaryFile(dir=".",delete=False).write(b"hello")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: write to closed file

'tempfile.NamedTemporaryFile' returns an instance of '_TemporaryFileWrapper' that wraps the created file object. The wrapper object implements '__getattr__' and we end up with a situation like the following simplified example where the wrapper gets destroyed before the looked up method is called:

$ cat test.py 
class S(object):
   def __init__(self):
      print("S.__init__")   def __del__(self):      print("S.__del__")
   def foo(self):
      print("S.foo")

class T(object):
   def __init__(self, s):
      self.s = s
      print("T.__init__")
   def __getattr__(self, name):
        s = self.__dict__['s']
        a = getattr(s, name)
        print("__getattr__('%s')" % name)
        if not isinstance(a, int):
            setattr(self, name, a)
        return a
   def __del__(self):
      print("T.__del__")
T(S()).foo()
$ ./python test.py 
S.__init__
T.__init__
__getattr__('foo')
T.__del__
S.foo
S.__del__

I am only new to Python, but...

Having looked at the code, I am surprised that the _TemporaryFileWrapper is not a subclass of "file". Surely that would be the "python" way, and would solve this problem, too? __getattr__ would no longer be needed. The opening of the file is within the library, so _os.open() could be replaced with file()... It would require rewriting/reorganising chunks of this code...

I don't see an obvious way of solving that. One thing I could think of is creating a wrapper for file method being returned from __getattr__ that holds reference to _TemporaryFileWrapper instance until the method gets called, please find a patch attached.

Yet one way is just remove __del__() and monkey-patch underlied file.

I was experimenting with something similar to what Jakub has. Rebinding the call the the wrapper seems like a simple and clean way to do it. Although, I wasn't absolutely sure whether this approach covers all cases.

The monkey-patched version breaks the auto-close on __del__ feature:

[meadori@li589-207 cpython]$ ./python 
Python 3.4.0a1+ (default:c41c68a18bb6, Sep  5 2013, 17:51:03) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tempfile
>>> tempfile.NamedTemporaryFile(dir=".",delete=False).write(b"hello")
__main__:1: ResourceWarning: unclosed file <_io.BufferedRandom name=4>
5

Also run the test suite with '-R :' and you will see tons of resource warnings.

Breaking auto-close is because now unlinking *and* closing are being guarded by 'delete'. Even if that is fixed you still run into resource leak problems. Looks like the monkey-patching introduces a reference cycle.

The wrapper binding approach works fine in all the tests I did.

The monkey-patched version breaks the auto-close on __del__ feature:

Files were closed. Just with a warning. I consider this as an enhancement. If you don't close file explicitly you will get a leak on non-refcounted Python implementations. It's true for both ordinal files so tempfiles. Perhaps we should add a warning in any __del__ method (even if we will commit Jakub's patch, this way LGTM too).

Hit this issue too, but with the "read" method. I think that providing a custom read() and write() methods in the wrapper class that override __getattr__ is a sufficient solution. These are the attributes most likely to be used as the "tail".

Jakub's patch has a bug:

>>> import tempfile
>>> f = tempfile.NamedTemporaryFile(dir=".",delete=False)
>>> write = f.write
>>> write
<function BufferedRandom.write at 0xb716b4f4>
>>> write2 = f.write
>>> write2
<built-in method write of _io.BufferedRandom object at 0xb721deac>
>>> del f
>>> write(b'foo')
3
>>> del write
>>> write2(b'bar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: write to closed file

Here is fixed Jakub's patch.

Unfortunately last patch causes resource leak on 3.3+.

$ ./python -Wall -m test.regrtest test_tempfile
[1/1] test_tempfile
1 test OK.
sys:1: ResourceWarning: gc: 5 uncollectable objects at shutdown; use gc.set_debug(gc.DEBUG_UNCOLLECTABLE) to list them

Here is a proper patch (for 3.3). Untested under Windows, but should work.

Oops, removed two debug lines.

LGTM. Thank you Antoine.

New changeset f3b7a76fb778 by Antoine Pitrou in branch '3.3':
Issue bpo-18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called.
http://hg.python.org/cpython/rev/f3b7a76fb778

New changeset d68ab2eb7a77 by Antoine Pitrou in branch 'default':
Issue bpo-18879: When a method is looked up on a temporary file, avoid closing the file before the method is possibly called.
http://hg.python.org/cpython/rev/d68ab2eb7a77

Fixed in 3.3 and 3.4. I think 2.7 is irrelevant at this point.

bpo-15002 uses this patch to fix a similar wrapping problem in urllib.

Also, this affects 2.7 as well and bpo-15002 does report the problem for 2.7. I'd like to get this fix backported. Would that be OK?

Is bpo-21579 fixed in that bug? too many letters..sorry...