This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: tempfile doesn't seem to play nicely with os.chdir on Windows
Type: crash Stage:
Components: Library (Lib) Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Gabriele Tornetta, christian.heimes, eryksun
Priority: normal Keywords:

Created on 2020-12-31 15:44 by Gabriele Tornetta, last changed 2022-04-11 14:59 by admin.

Files
File name Uploaded Description Edit
tempfile_st.txt Gabriele Tornetta, 2020-12-31 15:44 Stacktrace
Messages (5)
msg384125 - (view) Author: Gabriele N Tornetta (Gabriele Tornetta) * Date: 2020-12-31 15:44
The following script causes havoc on Windows while it works as expected on  Linux

~~~ python
import os
import tempfile


def test_chdir():
    with tempfile.TemporaryDirectory() as tempdir:
        os.chdir(tempdir)
~~~

Running the above on Windows results in RecursionError: maximum recursion depth exceeded while calling a Python object (see attachment for the full stacktrace).
msg384165 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2021-01-01 13:01
The code fails because TemporaryDirectory.__exit__() is unable to remove the directory. Windows doesn't let you remove files and directories that are used by a process. On POSIX-like operating systems like Linux support removing of opened files. For example anonymous temporary files use the trick.
msg384166 - (view) Author: Gabriele N Tornetta (Gabriele Tornetta) * Date: 2021-01-01 13:34
That makes sense, but I wonder what the "right" behaviour should be in this case. Surely the infinite recursion should be fixed at the very minimum. Perhaps the code on Windows could be enhanced to catch the case whereby one is trying to delete the cwd and do something like chdir('..') and then delete the temp folder. However, I suspect that something like this still wouldn't be enough. For example, this works fine

~~~ python
def test_chdir():
    with tempfile.TemporaryDirectory() as tempdir:
        old = os.getcwd()
        os.chdir(tempdir)
        os.chdir(old)
~~~

whereas this doesn't (same stacktrace as the original case)

~~~ python
def test_chdir():
    with tempfile.TemporaryDirectory() as tempdir:
        old = os.getcwd()
        os.chdir(tempdir)
        with open(os.path.join(tempdir, "delme")) as fout:
            fout.write("Hello")
        os.chdir(old)
~~~
msg384168 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-01 13:52
> Windows doesn't let you remove files and directories that are used 
> by a process. 

Windows does allow deleting open files/directories, but read/execute, write/append, and delete/rename access have to be explicitly shared when opening a file or directory. WinAPI SetCurrentDirectoryW (called by os.chdir) does not share delete access when opening a directory as the new working directory, and the handle is kept open to use as the NTAPI RootDirectory handle when opening relative paths. So the directory can only be deleted after either the working directory changes or the process exits.

See bpo-35144 for the source of the recursion error. An onerror handler was added that retries rmtree() after attempting to reset permissions. I proposed a workaround in the linked issue, but I don't think it's enough. Maybe there should be a separate onerror function for Windows.

The two common PermissionError cases to handle in Windows are readonly files and sharing violations. If the readonly attribute is set, we can remove it and retry the unlink() or rmdir() call. I don't think there's no need for a recursive call since rmtree() only removes empty directories. 

For a sharing violation (winerror 32), all that can be done is to set an atexit function that retries deleting the directory. If rmtree() still fails, emit a resource warning and give up.
msg384172 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-01 14:27
> def test_chdir():
>     with tempfile.TemporaryDirectory() as tempdir:
>         old = os.getcwd()
>         os.chdir(tempdir)
>         with open(os.path.join(tempdir, "delme")) as fout:
>             fout.write("Hello")
>         os.chdir(old)

The open() call in test_chdir() fails before os.chdir(old) executes. You would need to call os.chdir(old) in a `finally` block.

But the cleanup routine could still fail due to a sharing violation from some other process (likely a child process) setting the directory or a child directory as the working directory. Also, a sharing violation when trying to delete a regular file in the tree causes the cleanup routine to fail with NotADirectoryError because the onerror function calls rmtree() if unlink() fails with a PermissionError.
History
Date User Action Args
2022-04-11 14:59:39adminsetgithub: 86962
2021-01-01 14:27:04eryksunsetmessages: + msg384172
2021-01-01 13:52:53eryksunsetnosy: + eryksun
messages: + msg384168
2021-01-01 13:34:49Gabriele Tornettasetmessages: + msg384166
2021-01-01 13:01:52christian.heimessetnosy: + christian.heimes
messages: + msg384165
2020-12-31 15:44:22Gabriele Tornettacreate