This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: shutil.rmtree using vagrant synched folder fails
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: PeterFS, lkantola, miss-islington, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-01-14 09:45 by PeterFS, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
Vagrantfile PeterFS, 2020-01-14 09:45
Pull Requests
URL Status Linked Edit
PR 31384 merged python-dev, 2022-02-17 10:46
PR 31450 merged miss-islington, 2022-02-20 16:02
PR 31451 merged miss-islington, 2022-02-20 16:02
Messages (5)
msg359961 - (view) Author: Peter Liedholm (PeterFS) * Date: 2020-01-14 09:45
Python 3.6.9
Ubuntu 18.04
python3 -c 'import shutil; shutil.rmtree("1a")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 428, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 426, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 26] Text file busy: '4a'
-----------------
Reproduction method
mkdir synched_folder\1a\2a\3a\4a
mkdir synched_folder\1a\2b\3a\4a
mkdir synched_folder\1b\2a\3a\4a
choco install vagrant
Save Vagrantfile in empty folder
vagrant box add ubuntu/bionic64
vagrant up
vagrant ssh
cd synched_folder
python3 -c 'import shutil; shutil.rmtree("1a")'
*** Error message ***
rm -r 1a
*** Works fine ***
msg360028 - (view) Author: Peter Liedholm (PeterFS) * Date: 2020-01-15 07:13
Problem is also reported in virtualbox
https://www.virtualbox.org/ticket/19004

From that ticket some more analysis is done;
strace reveals that Python has kept an open fd for the directory being removed.
msg408174 - (view) Author: Lasse Kantola (lkantola) Date: 2021-12-10 05:02
I have encountered the same problem with vboxsf mounted folders when using Debian guest on VirtualBox 6.1 hosted by Windows 10.

The general VirtualBox issue of vboxsf not supporting rmdir when there are open file descriptors is difficult to solve, and out of scope for Python.

However, it looks like the shutil.rmtree implementation can be easily modified to change the order of operations so that the rmdir is performed after the directory file descriptor is closed, which makes shutil.rmtree work under vboxsf shared folders.

Example of the change against Python 3.7 is inlined below. The changed order matches the behavior of "rm -r" and to my understanding should not change the security implications of the code, but as the code is related to the prevention of a symlink attack https://bugs.python.org/issue4489 careful review would be needed.


--- /usr/lib/python3.7/shutil.py--orig  2021-01-23 05:04:44.000000000 +0900
+++ /usr/lib/python3.7/shutil.py        2021-12-09 20:42:36.795338346 +0900
@@ -427,10 +427,6 @@
                 try:
                     if os.path.samestat(orig_st, os.fstat(dirfd)):
                         _rmtree_safe_fd(dirfd, fullname, onerror)
-                        try:
-                            os.rmdir(entry.name, dir_fd=topfd)
-                        except OSError:
-                            onerror(os.rmdir, fullname, sys.exc_info())
                     else:
                         try:
                             # This can only happen if someone replaces
@@ -442,6 +438,10 @@
                             onerror(os.path.islink, fullname, sys.exc_info())
                 finally:
                     os.close(dirfd)
+                try:
+                    os.rmdir(entry.name, dir_fd=topfd)
+                except OSError:
+                    onerror(os.rmdir, fullname, sys.exc_info())
         else:
             try:
                 os.unlink(entry.name, dir_fd=topfd)
@@ -489,10 +489,6 @@
         try:
             if os.path.samestat(orig_st, os.fstat(fd)):
                 _rmtree_safe_fd(fd, path, onerror)
-                try:
-                    os.rmdir(path)
-                except OSError:
-                    onerror(os.rmdir, path, sys.exc_info())
             else:
                 try:
                     # symlinks to directories are forbidden, see bug #1669
@@ -501,6 +497,10 @@
                     onerror(os.path.islink, path, sys.exc_info())
         finally:
             os.close(fd)
+        try:
+            os.rmdir(path)
+        except OSError:
+            onerror(os.rmdir, path, sys.exc_info())
     else:
         try:
             if os.path.islink(path):
msg413595 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-02-20 16:02
New changeset b77158b4da449ec5b8f682816a79d004fd65ed07 by Lital Natan in branch 'main':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
https://github.com/python/cpython/commit/b77158b4da449ec5b8f682816a79d004fd65ed07
msg413636 - (view) Author: miss-islington (miss-islington) Date: 2022-02-21 08:08
New changeset 959846be52b03da57f50ebc07f99ec262a86e860 by Miss Islington (bot) in branch '3.9':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
https://github.com/python/cpython/commit/959846be52b03da57f50ebc07f99ec262a86e860
msg413637 - (view) Author: miss-islington (miss-islington) Date: 2022-02-21 08:08
New changeset 95d6271f1997580f8cb8a2985d0cd48af912763f by Miss Islington (bot) in branch '3.10':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
https://github.com/python/cpython/commit/95d6271f1997580f8cb8a2985d0cd48af912763f
History
Date User Action Args
2022-04-11 14:59:25adminsetgithub: 83508
2022-02-21 09:19:35serhiy.storchakasetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: + Python 3.9, Python 3.10, Python 3.11, - Python 3.6
2022-02-21 08:08:00miss-islingtonsetmessages: + msg413637
2022-02-21 08:08:00miss-islingtonsetmessages: + msg413636
2022-02-20 16:02:54miss-islingtonsetpull_requests: + pull_request29580
2022-02-20 16:02:51miss-islingtonsetnosy: + miss-islington
pull_requests: + pull_request29579
2022-02-20 16:02:24serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg413595
2022-02-17 10:46:59python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request29532
stage: patch review
2021-12-10 05:02:34lkantolasetnosy: + lkantola
messages: + msg408174
2020-01-15 07:13:06PeterFSsetmessages: + msg360028
2020-01-14 09:45:52PeterFScreate