classification
Title: shutil.rmtree should not fail with FileNotFoundError (race condition)
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: dkg, giampaolo.rodola, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-03-02 19:12 by dkg, last changed 2018-07-11 14:32 by serhiy.storchaka.

Files
File name Uploaded Description Edit
breaker.c dkg, 2017-03-02 19:12 C program to tickle the shutil.rmtree race condition
demo.py dkg, 2017-03-02 19:13
Messages (4)
msg288822 - (view) Author: Daniel Kahn Gillmor (dkg) Date: 2017-03-02 19:12
There is a race condition in shutil.rmtree, where if a file gets removed between when rmtree plans to remove it and when it gets around to removing it, a FileNotFound exception gets raised.

The expected semantics of rmtree imply that if the filesystem tree is removed, then the command has succeeded, so it doesn't make sense for rmtree to raise a FileNotFound error if someone else happened to have deleted the file before rmtree gets to it.

I'm attaching a C program (for GNU/Linux) which uses inotify to remove the other file in a directory when either file is removed.  This triggers the rmtree failure.

This behavior has caused a number of workarounds in external projects, like:

  https://bitbucket.org/vinay.sajip/python-gnupg/commits/492fd45ca073a90aac434320fb0c8fe8d01f782b
  https://git.gnupg.org/cgi-bin/gitweb.cgi?p=gpgme.git;a=commitdiff;h=de8494b16bc50c60a8438f2cae1f8c88e8949f7a

It would be better for shutil.rmtree to ignore this particular exception (FileNotFoundError).

Another option for users is to set ignore_errors=True, but this ends up ignoring *all* errors, which doesn't seem like the right decision.

Finally, of course, a user could specify some sort of onerror function that explictly ignores FileNotFoundError, but this seems pretty complicated for the common pattern.

It's possible that shutil.rmtree() wants to raise FileNotFoundError if the actual argument passed by the user does not itself exist, but it really doesn't make sense to raise that error for any of the elements further down in the tree.
msg288823 - (view) Author: Daniel Kahn Gillmor (dkg) Date: 2017-03-02 19:13
and here is python demonstration script that will build breaker.c and then use it to cause the error to be raised from shutils.rmtree.

the output of demo.py looks like this:

make: 'breaker' is up to date.
Traceback (most recent call last):
  File "./demo.py", line 14, in <module>
    shutil.rmtree('xx')
  File "/usr/lib/python3.5/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.5/shutil.py", line 438, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/usr/lib/python3.5/shutil.py", line 436, in _rmtree_safe_fd
    os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: 'b'
msg319374 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-06-12 10:20
+1. It looks reasonable to ignore FileNotFoundError on os.rmdir(),  os.unlink() and also os.open() and os.scandir().
msg321458 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-07-11 14:32
Shouldn't this be considered as a new feature? There are no guaranties that shutil.rmtree() should work if files or directories are concurrently removed, or created, or made read-only in other threads or processes.
History
Date User Action Args
2018-07-11 14:32:37serhiy.storchakasetmessages: + msg321458
2018-07-11 14:23:10vstinnersetnosy: + serhiy.storchaka
2018-07-11 07:55:24serhiy.storchakasettype: crash -> behavior
2018-06-12 10:20:29giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg319374
2017-03-02 19:13:29dkgsetfiles: + demo.py

messages: + msg288823
2017-03-02 19:12:24dkgcreate