classification
Title: tempfile.TemporaryDirectory() cleanup exception if nonwriteable or non-searchable files or directories created
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Laurent.Mazuel, altendky, eryksun, georg.brandl, masthana, rtzoeller, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2016-03-28 22:22 by Laurent.Mazuel, last changed 2019-05-31 08:30 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 10320 merged serhiy.storchaka, 2018-11-04 14:44
Messages (7)
msg262584 - (view) Author: Laurent Mazuel (Laurent.Mazuel) Date: 2016-03-28 22:22
Using tempfile.TemporaryDirectory() in Windows, creating read-only files in this temp directory leads to PermissionError during the cleanup().
This is a direct cause of this one:
https://bugs.python.org/issue19643

And the workaround which was proposed in the issue 19643 and added to the doc here:
https://docs.python.org/3/library/shutil.html?highlight=shutil#rmtree-example

is not used in the TemporaryDirectory implementation.

I don't know if the right solution is to modify the implementation to systematically delete read-only files using the cited workaround, or to add a 'remove_readonly' flag or to update the documentation to clearly says that cleanup will raise a PermissionError if the user creates a read-only file. At least documentation please :)

In my specific usecase I "git clone" in the temp directory, and the .git folder contains read-only files.

Full stacktrace:
Traceback (most recent call last):
  File "C:\mycode.py", line 149, in build_libraries
    update(generated_path, dest_folder)
  File "C:\Program Files\Python35\lib\tempfile.py", line 807, in __exit__
    self.cleanup()
  File "C:\Program Files\Python35\lib\tempfile.py", line 811, in cleanup
    _shutil.rmtree(self.name)
  File "C:\Program Files\Python35\lib\shutil.py", line 488, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Program Files\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python35\lib\shutil.py", line 378, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\Program Files\Python35\lib\shutil.py", line 383, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Program Files\Python35\lib\shutil.py", line 381, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\me\\AppData\\Local\\Temp\\tmpk62cp34t\\readonly.file'
msg329153 - (view) Author: Mayank Asthana (masthana) * Date: 2018-11-02 19:29
I think it is OK to delete read-only files since they are in a TemporaryDirectory and should be expected to be temporary. A note in the documentation about this behaviour should be sufficient.
msg329186 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-03 06:54
I am working on this. Left to test on Windows and analyze possible security issues.
msg329206 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-11-03 20:16
Serhiy, do you also plan to work around immutable files in POSIX? What about permissions on directories that prevent deleting files? 

In BSD and macOS, we have os.chflags for modifying file flags. Modifying the immutable flag requires superuser access. In Linux, modifying the immutable attribute requires the CAP_LINUX_IMMUTABLE capability, and file attributes are accessed with an ioctl call. For example: 

    import os
    import fcntl
    import array

    EXT2_IOC_GETFLAGS = 0x80086601
    EXT2_IOC_SETFLAGS = 0x40086602
    EXT2_IMMUTABLE_FL = 0x00000010 

    def make_mutable(filename):
        flags = array.array('l', [0])
        fd = os.open(filename, os.O_RDONLY)
        try:
            fcntl.ioctl(fd, EXT2_IOC_GETFLAGS, flags, True)
            flags[0] &= ~EXT2_IMMUTABLE_FL
            fcntl.ioctl(fd, EXT2_IOC_SETFLAGS, flags)
        finally:
            os.close(fd)

I assume for Windows this will use os.chmod. I need to rant a bit to see whether anyone else is bothered by this. Microsoft's chmod function modifies the readonly attribute as if it's a write/delete permission. I wish Python hadn't adopted this behavior, since it clashes with how chmod works on other platforms, none of which conflates native file attributes and permissions. (We can be granted permission to modify or delete an immutable file, but this doesn't enable us to modify the file until it's made mutable.) It would be nice to have os.get_file_attributes and os.set_file_attributes on Windows instead of this confused use of chmod.
msg329232 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-11-04 14:53
PR 10320 solves problems with file mode and flags which can be set by os.chmod() and os.chflags() by unprivileged user. Was tested on Linux, FreeBSD and Windows.

It doesn't solve problem when files was made immutable by chattr on Linux.
msg329236 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-11-04 16:41
> file mode and flags which can be set by os.chmod() and os.chflags() 
> by unprivileged user. 

I remembered incorrectly about chflags. It's not a single immutable flag, but several (depending on the OS). The owner or superuser can modify UF_IMMUTABLE, UF_READONLY, and UF_NOUNLINK. The SF_* flags can only be unset by the superuser in single-user mode.

> It doesn't solve problem when files was made immutable by chattr on
> Linux.

We may not have CAP_LINUX_IMMUTABLE, but we can at least try to clear the immutable flag, no?
msg344037 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2019-05-31 08:30
New changeset e9b51c0ad81da1da11ae65840ac8b50a8521373c by Serhiy Storchaka in branch 'master':
bpo-26660, bpo-35144: Fix permission errors in TemporaryDirectory cleanup. (GH-10320)
https://github.com/python/cpython/commit/e9b51c0ad81da1da11ae65840ac8b50a8521373c
History
Date User Action Args
2019-05-31 08:30:40serhiy.storchakasetmessages: + msg344037
2018-11-19 14:02:10rtzoellersetnosy: + rtzoeller
2018-11-04 16:41:55eryksunsetmessages: + msg329236
2018-11-04 14:53:49serhiy.storchakasetmessages: + msg329232
2018-11-04 14:44:14serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request9621
2018-11-04 14:42:56serhiy.storchakasettitle: tempfile.TemporaryDirectory() cleanup exception on Windows if readonly files created -> tempfile.TemporaryDirectory() cleanup exception if nonwriteable or non-searchable files or directories created
versions: + Python 3.7, Python 3.8, - Python 3.5
2018-11-03 20:16:29eryksunsetnosy: + eryksun
messages: + msg329206
2018-11-03 06:54:26serhiy.storchakasetmessages: + msg329186
2018-11-02 19:29:57masthanasetnosy: + masthana
messages: + msg329153
2018-11-02 12:23:57serhiy.storchakasetassignee: serhiy.storchaka

nosy: + serhiy.storchaka
2016-10-24 19:52:46altendkysetnosy: + altendky
2016-03-29 08:11:19SilentGhostsetnosy: + georg.brandl

versions: + Python 3.6
2016-03-28 22:22:28Laurent.Mazuelcreate