classification
Title: tempfile._get_default_tempdir() leaves files behind when HD is full
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: georg.brandl, kichik, ncoghlan, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2012-12-28 05:37 by kichik, last changed 2013-02-12 23:06 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
fix_tempfile_leaving_files_behind.patch kichik, 2012-12-28 05:37 Suggested patch against today's tip 81121:6e2e5adc0400 review
fix_tempfile_leaving_files_behind_2.patch serhiy.storchaka, 2013-02-05 19:17 Patch for 3.3+ review
fix_tempfile_leaving_files_behind_2-2.7.patch serhiy.storchaka, 2013-02-05 19:28 review
fix_tempfile_leaving_files_behind_2-3.2.patch serhiy.storchaka, 2013-02-05 19:28 review
Messages (9)
msg178362 - (view) Author: Amir Szekely (kichik) * Date: 2012-12-28 05:37
tempfile._get_default_tempdir() tries to find a good temporary directory by attempting to create a file and write a string into it for all candidate directories. It deletes those files right after closing them. But if writing rather than creating them fails, the files get left behind. This can happen, for example, when the disk is full on Linux. It will let you create the file with open() and even write() to it. But as soon as close() is called, OSError is raised with errno 28 (No space left on device) and unlink() is never called.

On our system (Linux 2.6.32, ext3), after filling the disk by mistake, we suddenly found dozens of random empty files scattered across the file system. The candidate list, where they were all found, even includes the root directory so the issue is quite visible.

We noticed this problem on Python 2.4.4, but I have been able to reproduce it with today's tip.

A possible solution would be wrapping everything between open() and unlink() in a try-finally block. This way, no matter what happens, the file will get deleted. I have attached a patch implementing this solution and a new test case that reproduces the issue. Verified with `make test && make patchcheck`.
msg178365 - (view) Author: Amir Szekely (kichik) * Date: 2012-12-28 06:11
Reproduction details:

[root@meh /]# uname -a
Linux meh 2.6.32-131.17.1.el6-0.0.3.smp.pae.gcc4.1.x86.i686 #1 SMP Tue Dec 20 02:40:22 PST 2011 i686 GNU/Linux
[root@meh /]# pwd
/
[root@meh /]# dd if=/dev/zero of=/tmp/BIG bs=4M
dd: writing `/tmp/BIG': No space left on device
31+0 records in
30+0 records out
127254528 bytes (127 MB) copied, 3.7028 s, 34.4 MB/s
[root@meh /]# ls
bin  boot  dev  etc  home  initrd  lib  lost+found  media  mnt  opt  proc  root  sbin  srv  sys  tmp  usr  var
[root@meh /]# python -c "import tempfile; tempfile.mktemp()"
Traceback (most recent call last):
  File "<string>", line 1, in ?
  File "/usr/lib/python2.4/tempfile.py", line 362, in mktemp
    dir = gettempdir()
  File "/usr/lib/python2.4/tempfile.py", line 268, in gettempdir
    tempdir = _get_default_tempdir()
  File "/usr/lib/python2.4/tempfile.py", line 214, in _get_default_tempdir
    raise IOError, (_errno.ENOENT,
IOError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
[root@meh /]# ls
bin  boot  dev  etc  home  initrd  lib  lost+found  media  mnt  opt  proc  -qE-nI  root  sbin  srv  sys  tmp  usr  var
[root@meh /]#
msg181469 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-02-05 19:17
In general it looks good to me. But we can get rid of Python file and use only os.open/os.write/os.close. Here is an updated patch which carefully close a file descriptor and remove a file even if write or close failed.

Amir Szekely, can you please submit a contributor form?

http://python.org/psf/contrib/contrib-form/
http://python.org/psf/contrib/
msg181471 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-02-05 19:28
And here are patches for 2.7 and 3.2 (there are some peculiarities, i.e. in 
exception handling).
msg181498 - (view) Author: Amir Szekely (kichik) * Date: 2013-02-06 05:02
I've submitted a contribution form by e-mail.
msg181587 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-07 09:45
There are reasons to use buffered I/O rather than os.write: os.write can fail with EINTR, for example.
msg181981 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-12 22:42
New changeset b368fc93dca8 by Serhiy Storchaka in branch '2.7':
Issue #16800: tempfile.gettempdir() no longer left temporary files when
http://hg.python.org/cpython/rev/b368fc93dca8

New changeset 377123f10820 by Serhiy Storchaka in branch '3.2':
Issue #16800: tempfile.gettempdir() no longer left temporary files when
http://hg.python.org/cpython/rev/377123f10820

New changeset 6f432bb11b28 by Serhiy Storchaka in branch '3.3':
Issue #16800: tempfile.gettempdir() no longer left temporary files when
http://hg.python.org/cpython/rev/6f432bb11b28

New changeset b66a5b41d82f by Serhiy Storchaka in branch 'default':
Issue #16800: tempfile.gettempdir() no longer left temporary files when
http://hg.python.org/cpython/rev/b66a5b41d82f
msg181982 - (view) Author: Roundup Robot (python-dev) Date: 2013-02-12 23:02
New changeset a43f67e95ef0 by Serhiy Storchaka in branch '2.7':
Fix for issue #16800: Use buffered write to handle EINTR.
http://hg.python.org/cpython/rev/a43f67e95ef0

New changeset 4622206db91b by Serhiy Storchaka in branch '3.2':
Fix for issue #16800: Use buffered write to handle EINTR.
http://hg.python.org/cpython/rev/4622206db91b

New changeset 2fb03fe354e3 by Serhiy Storchaka in branch '3.3':
Fix for issue #16800: Use buffered write to handle EINTR.
http://hg.python.org/cpython/rev/2fb03fe354e3

New changeset fec33725f319 by Serhiy Storchaka in branch 'default':
Fix for issue #16800: Use buffered write to handle EINTR.
http://hg.python.org/cpython/rev/fec33725f319
msg181983 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-02-12 23:06
Thank you for explanation, Antoine.

Thank you for your contribution, Amir.
History
Date User Action Args
2013-02-12 23:06:03serhiy.storchakasetstatus: open -> closed
messages: + msg181983

assignee: serhiy.storchaka
resolution: fixed
stage: patch review -> resolved
2013-02-12 23:02:26python-devsetmessages: + msg181982
2013-02-12 22:42:25python-devsetnosy: + python-dev
messages: + msg181981
2013-02-07 09:45:47pitrousetnosy: + pitrou
messages: + msg181587
2013-02-06 05:02:22kichiksetmessages: + msg181498
2013-02-05 19:28:37serhiy.storchakasetfiles: + fix_tempfile_leaving_files_behind_2-2.7.patch, fix_tempfile_leaving_files_behind_2-3.2.patch

messages: + msg181471
2013-02-05 19:20:31serhiy.storchakasetfiles: - fix_tempfile_leaving_files_behind_2.patch
2013-02-05 19:20:24serhiy.storchakasetmessages: - msg181470
2013-02-05 19:19:42serhiy.storchakasetfiles: + fix_tempfile_leaving_files_behind_2.patch

messages: + msg181470
2013-02-05 19:17:07serhiy.storchakasetfiles: + fix_tempfile_leaving_files_behind_2.patch
nosy: + georg.brandl, ncoghlan, serhiy.storchaka
messages: + msg181469

2012-12-28 06:23:18serhiy.storchakasetstage: patch review
components: + Library (Lib), - Extension Modules
versions: - Python 2.6, Python 3.1, Python 3.5
2012-12-28 06:11:48kichiksetmessages: + msg178365
2012-12-28 05:37:37kichikcreate