classification
Title: NamedTemporaryFile can leave temporary files behind
Type: behavior Stage: needs patch
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder: with statements are not ensuring that __exit__ is called if __enter__ succeeds
View: 29988
Assigned To: Nosy List: giampaolo.rodola, jwilk, ncoghlan, r.david.murray, rhettinger, xtreak
Priority: normal Keywords:

Created on 2018-07-19 19:29 by jwilk, last changed 2018-07-22 00:32 by giampaolo.rodola.

Messages (6)
msg321965 - (view) Author: Jakub Wilk (jwilk) Date: 2018-07-19 19:29
If you press Ctrl+C at the wrong moment, NamedTemporaryFile won't delete the
temporary file. To reproduce, you can try this script:

    import tempfile
    while True:
        with tempfile.NamedTemporaryFile(dir='.'):
            pass

I get a stray temporary file when I press Ctrl+C about 40% of the time.
msg321974 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-07-20 00:44
Nick, is this related to the bug where the "finally" portion of a context manager isn't guaranteed to be called?
msg321999 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-07-20 10:56
Aye, this is a specific case of the general issue noted in https://bugs.python.org/issue29988 (while it may be partially mitigated by https://bugs.python.org/issue32949, it isn't a guarantee)
msg322009 - (view) Author: Jakub Wilk (jwilk) Date: 2018-07-20 13:49
I think issue29988 is unrelated, or at least not the whole story.

These are samples of tracebacks I see when the file is left behind:

  Traceback (most recent call last):
    File "test-tmpfile.py", line 4, in <module>
      with tempfile.NamedTemporaryFile(dir='.'):
    File "/opt/python/lib/python3.8/tempfile.py", line 548, in NamedTemporaryFile
      (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
    File "/opt/python/lib/python3.8/tempfile.py", line 258, in _mkstemp_inner
      fd = _os.open(file, flags, 0o600)
  KeyboardInterrupt
  
  Traceback (most recent call last):
    File "test-tmpfile.py", line 4, in <module>
      with tempfile.NamedTemporaryFile(dir='.'):
    File "/opt/python/lib/python3.8/tempfile.py", line 548, in NamedTemporaryFile
      (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
    File "/opt/python/lib/python3.8/tempfile.py", line 269, in _mkstemp_inner
      return (fd, _os.path.abspath(file))
    File "/opt/python/lib/python3.8/posixpath.py", line 371, in abspath
      path = os.fspath(path)
  KeyboardInterrupt
  
  Traceback (most recent call last):
    File "test-tmpfile.py", line 4, in <module>
      with tempfile.NamedTemporaryFile(dir='.'):
    File "/opt/python/lib/python3.8/tempfile.py", line 548, in NamedTemporaryFile
      (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
    File "/opt/python/lib/python3.8/tempfile.py", line 269, in _mkstemp_inner
      return (fd, _os.path.abspath(file))
    File "/opt/python/lib/python3.8/posixpath.py", line 378, in abspath
      return normpath(path)
    File "/opt/python/lib/python3.8/posixpath.py", line 355, in normpath
      if comp in (empty, dot):
  KeyboardInterrupt

In all cases the interrupt happened in the NamedTemporaryFile function,
so before __enter__/__exit__ would have chance to do its job.
msg322013 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2018-07-20 15:17
It's still the same problem - the underlying issue is that the with statement machinery doesn't currently mask signals in the main thread while __enter__ and __exit__ are running, so __enter__ and __exit__ methods written in Python are also at risk of being interrupted at an inopportune point.

This means that even "with closing(open('filename')) as f: ..." is more at risk of leaving the file open until __del__ cleans it up than the version that calls open() directly.

So if this a concern that an application needs to worry about, then it currently needs to adopt a more complicated execution structure where:

1. The main thread launches a subthread that actually does all the work.
2. The main thread immediately drops into "active_thread.join()" (which can be interrupted by Ctrl-C)

Unfortunately, this scheme *doesn't* work for applications where the application itself needs to detect and handling signals other than Ctrl-C.
msg322027 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2018-07-20 16:35
Nick, what Jakub is saying is that 'with' hasn't even gotten involved yet: we're still executing the NamedTemporaryFile constructor, so the object hasn't been returned for 'with' to operate on yet.  In other words, NamedTemporaryFile.__init__ isn't safe against ctl-C when it calls _mkstemp, which is obvious by inspection since it isn't inside the try/except.
History
Date User Action Args
2018-07-22 00:32:12giampaolo.rodolasetnosy: + giampaolo.rodola
2018-07-21 06:55:06xtreaksetnosy: + xtreak
2018-07-20 16:36:19r.david.murraysetstatus: closed -> open
resolution: duplicate ->
stage: resolved -> needs patch
2018-07-20 16:35:12r.david.murraysetnosy: + r.david.murray
messages: + msg322027
2018-07-20 15:17:51ncoghlansetmessages: + msg322013
2018-07-20 13:49:52jwilksetmessages: + msg322009
2018-07-20 10:56:36ncoghlansetstatus: open -> closed
superseder: with statements are not ensuring that __exit__ is called if __enter__ succeeds
messages: + msg321999

resolution: duplicate
stage: resolved
2018-07-20 00:44:48rhettingersetnosy: + rhettinger, ncoghlan
messages: + msg321974
2018-07-19 19:29:32jwilkcreate