classification
Title: TemporaryDirectory clean-up fails with unsearchable directories
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: lilydjwg, serhiy.storchaka, xtreak
Priority: normal Keywords: patch

Created on 2018-11-02 08:40 by lilydjwg, last changed 2019-05-31 08:30 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 10320 merged serhiy.storchaka, 2018-11-04 14:44
Messages (5)
msg329116 - (view) Author: lilydjwg (lilydjwg) * Date: 2018-11-02 08:40
If the title doesn't explain clearly, here's a demo program that will fail:

import tempfile
import pathlib

def test():
  with tempfile.TemporaryDirectory(prefix='test-bad-') as tmpdir:
    tmpdir = pathlib.Path(tmpdir)
    subdir = tmpdir / 'sub'
    subdir.mkdir()
    with open(subdir / 'file', 'w'):
      pass
    subdir.chmod(0o600)

if __name__ == '__main__':
  test()

I didn't expect this, and I didn't find an easy way to handle this except not using TemporaryDirectory at all:

import tempfile
import pathlib
import shutil
import os

def rmtree_error(func, path, excinfo):
  if isinstance(excinfo[1], PermissionError):
    os.chmod(os.path.dirname(path), 0o700)
    os.unlink(path)
  print(func, path, excinfo)

def test():
  tmpdir = tempfile.mkdtemp(prefix='test-good-')
  try:
    tmpdir = pathlib.Path(tmpdir)
    subdir = tmpdir / 'sub'
    subdir.mkdir()
    with open(subdir / 'file', 'w'):
      pass
    subdir.chmod(0o600)
  finally:
    shutil.rmtree(tmpdir, onerror=rmtree_error)

if __name__ == '__main__':
  test()

This works around the issue, but the dirfd is missing in the onerror callback.

I have this issue because my program extracts tarballs to a temporary directory for examination. I expected that TemporaryDirectory cleaned up things when it could.

What do you think? rm -rf can't remove such a directory either but this is annoying and I think Python can do better.
msg329127 - (view) Author: Karthikeyan Singaravelan (xtreak) * (Python committer) Date: 2018-11-02 11:41
Seems like a related issue : issue26660 . Maybe TemporaryDirectory can allow an onerror argument that is passed internally to rmtree during cleanup and state the same in the documentation that TemporaryDirectory can't cleanup read-only files?
msg329184 - (view) Author: lilydjwg (lilydjwg) * Date: 2018-11-03 06:50
Yes issue26660 is similar but not the same. On Windows it seems a read-only file cannot be deleted while on Linux a file resident in a non-searchable directory cannot be deleted.

An onerror for TemporaryDirectory will work. Also I'd like to add an optional dir_fd argument for onerror.
msg329185 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-03 06:52
I am working on this. Left to test on Windows and analyze possible security issues.
msg344036 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-31 08:30
New changeset e9b51c0ad81da1da11ae65840ac8b50a8521373c by Serhiy Storchaka in branch 'master':
bpo-26660, bpo-35144: Fix permission errors in TemporaryDirectory cleanup. (GH-10320)
https://github.com/python/cpython/commit/e9b51c0ad81da1da11ae65840ac8b50a8521373c
History
Date User Action Args
2019-05-31 08:30:40serhiy.storchakasetmessages: + msg344036
2018-11-04 14:44:15serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request9622
2018-11-03 06:52:53serhiy.storchakasetmessages: + msg329185
2018-11-03 06:50:33lilydjwgsetmessages: + msg329184
2018-11-02 20:32:05terry.reedysettitle: TemporaryDirectory can't be cleaned up if there are unsearchable directories -> TemporaryDirectory clean-up fails with unsearchable directories
2018-11-02 12:24:05serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
2018-11-02 11:41:10xtreaksetmessages: + msg329127
2018-11-02 11:19:05xtreaksetnosy: + xtreak
2018-11-02 08:40:08lilydjwgcreate