classification
Title: shutil.rmtree fails if inner folder is open in Windows Explorer
Type: behavior Stage: test needed
Components: IO, Library (Lib), Windows Versions: Python 3.8, Python 3.7, Python 3.6
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, giampaolo.rodola, paul.moore, steve.dower, tarek, tim.golden, yuliu, zach.ware
Priority: normal Keywords:

Created on 2018-04-08 02:51 by yuliu, last changed 2018-04-13 19:48 by terry.reedy.

Messages (8)
msg315077 - (view) Author: Yu Liu (yuliu) Date: 2018-04-08 02:51
Given the following directory structure on a Windows machine:
- foo
 - bar

a call to `shutil.rmtree("foo")` will fail when the inner folder `bar` is opened in an Explorer. The error message indicates the `foo` directory is not empty, while after the execution, although it failed, the `foo` directory is empty. So the inner folder `bar` was removed successfully, but `foo` was not. And the error message is misleading.

It will not fail when `foo` is opened in an Explorer, neither on Linux system.
msg315078 - (view) Author: Eryk Sun (eryksun) * Date: 2018-04-08 04:35
This is not an uncommon problem. If there's one or more existing references to a file or empty directory that were opened with shared delete access, then a delete operation will succeed, but the file or directory will not be unlinked from the parent directory. The filesystem only unlinks a file or directory if its "delete disposition" is set when the last reference is closed. 

Removing the parent directory thus requires a loop that retries the delete until it succeeds, i.e. until existing references are closed and the directory finally becomes empty and thus deletable. If the problem is caused by an anti-malware program, it should typically be resolved within a short time. Exactly how long to wait in a retry loop before failing the operation should be configurable.

Maybe you can conduct a simple experiment to measure the wait time required in your case. Run the following with "bar" opened in Explorer. Substitute the real path of "foo" in PARENT_PATH.

    import os
    import time

    ERROR_DIR_NOT_EMPTY = 145

    PARENT_PATH = 'foo'
    CHILD_PATH = os.path.join(PARENT_PATH, 'bar')

    os.rmdir(CHILD_PATH)
    t0 = time.perf_counter()

    while True:
        try:
            os.rmdir(PARENT_PATH)
            wait_time = time.perf_counter() - t0
            break
        except OSError as e:
            if e.winerror != ERROR_DIR_NOT_EMPTY:
               raise

    print(wait_time)
msg315103 - (view) Author: Yu Liu (yuliu) Date: 2018-04-09 01:13
The result is 0.00026412295632975946 on my computer. Does this mean it is caused by an anti-malware program?
msg315105 - (view) Author: Eryk Sun (eryksun) * Date: 2018-04-09 02:41
A sub-millisecond wait is fairly quick, but it depends on the machine speed. I should have included a counter. Try the following. It's not reproducing the problem if num_retries doesn't get incremented.

    import os
    import time

    ERROR_DIR_NOT_EMPTY = 145

    PARENT_PATH = 'foo'
    CHILD_PATH = os.path.join(PARENT_PATH, 'bar')

    os.rmdir(CHILD_PATH)

    num_retries = 0
    t0 = time.perf_counter()

    while True:
        try:
            os.rmdir(PARENT_PATH)
            break
        except OSError as e:
            if e.winerror != ERROR_DIR_NOT_EMPTY:
               raise
            num_retries += 1

    wait_time = time.perf_counter() - t0

    print('num_retries:', num_retries)
    print('wait_time:', wait_time)
msg315110 - (view) Author: Yu Liu (yuliu) Date: 2018-04-09 07:02
This time I run it a couple of consecutive times manually. The result shows that the first time it retried 12 times and the wait time was 0.0004259171330071893. Then it seems to be steady, and the num_retried is 3 or 4, and the wait_time is about 0.00025.

I also run it without `bar` opened in Explorer. The num_retried is 0 and the wait_time is 4.46745114706336e-05.

Because of some reasons, I can't post the exact results here. I will run it on another computer later and post the results if you need all of the exact results.
msg315126 - (view) Author: Yu Liu (yuliu) Date: 2018-04-09 12:02
These are results on another slower machine. Note these results are attained on Windows 10, while the above on Windows 7. Just in case it has some influence.

    $ python test.py
    num_retries: 6
    wait_time: 0.0008691726957804373
    $ python test.py
    num_retries: 3
    wait_time: 0.0007175661796639806
    $ python test.py
    num_retries: 6
    wait_time: 0.0007962191842657514
    $ python test.py
    num_retries: 3
    wait_time: 0.0006970480045504753
    $ python test.py
    num_retries: 4
    wait_time: 0.0009637842810260455
    $ python test.py
    num_retries: 4
    wait_time: 0.001005390580561765
    $ python test.py
    num_retries: 3
    wait_time: 0.000654301806397339
    $ python test.py
    num_retries: 6
    wait_time: 0.0008857012257329832
    $ python test.py
    num_retries: 4
    wait_time: 0.0009227479307990348
    $ python test.py
    num_retries: 4
    wait_time: 0.0008976701612158615

And this is result without `bar` opened in Explorer.

    $ python test.py
    num_retries: 0
    wait_time: 0.00019834235943055225
msg315129 - (view) Author: Eryk Sun (eryksun) * Date: 2018-04-09 15:11
Your case probably isn't due to a anti-malware filesystem filter. Explorer keeps handles open to directories to get updates via ReadDirectoryChangesExW. It opens watched directories with shared delete  access, so deleting the child succeeds. But as discussed above, the directory isn't unlinked from the parent until Explorer closes its handle. Apparently it's not fast enough on the systems you tested.

As a workaround, you can define an onerror handler for use with shutil.rmtree() that retries the rmdir() call in a loop for up to a given timeout period, such as 10 ms. For convenience, a handler that retries unlink() and rmdir() could be distributed with shutil. For ease of use, it could be enabled by default on Windows.
msg315185 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-04-11 07:19
> For convenience, a handler that retries unlink() and rmdir() could be distributed with shutil. For ease of use, it could be enabled by default on Windows.

+1 on that. I bumped into this many times over the years as occasional and hardly reproducible test failures when cleaning up test files/dirs. 
The tricky part is how to distinguish a legitimate "directory is not empty" error though.
History
Date User Action Args
2018-04-13 19:48:13terry.reedysettitle: shutil.rmtree fails when the inner floder is opened in Explorer on Windows -> shutil.rmtree fails if inner folder is open in Windows Explorer
2018-04-11 07:19:04giampaolo.rodolasetmessages: + msg315185
2018-04-09 15:11:35eryksunsetmessages: + msg315129
2018-04-09 12:02:08yuliusetmessages: + msg315126
2018-04-09 07:02:02yuliusetmessages: + msg315110
2018-04-09 02:41:52eryksunsetmessages: + msg315105
2018-04-09 01:13:14yuliusetmessages: + msg315103
2018-04-08 04:35:51eryksunsetversions: + Python 3.7, Python 3.8
nosy: + eryksun

messages: + msg315078

components: + IO
stage: test needed
2018-04-08 02:51:56yuliucreate