Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

shutil.rmtree using vagrant synched folder fails #83508

Closed
PeterFS mannequin opened this issue Jan 14, 2020 · 6 comments
Closed

shutil.rmtree using vagrant synched folder fails #83508

PeterFS mannequin opened this issue Jan 14, 2020 · 6 comments
Labels
3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@PeterFS
Copy link
Mannequin

PeterFS mannequin commented Jan 14, 2020

BPO 39327
Nosy @serhiy-storchaka, @miss-islington, @lkantola
PRs
  • bpo-39327: Fixed issue: Avoid "Text File Busy" OSError #31384
  • [3.9] bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384) #31450
  • [3.10] bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384) #31451
    		•
    Files
  • Vagrantfile
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2022-02-21.09:19:35.883>
created_at = <Date 2020-01-14.09:45:52.878>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'shutil.rmtree using vagrant synched folder fails'
updated_at = <Date 2022-02-21.09:19:35.878>
user = 'https://bugs.python.org/PeterFS'

    bugs.python.org fields:

    activity = <Date 2022-02-21.09:19:35.878>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-21.09:19:35.883>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2020-01-14.09:45:52.878>
creator = 'PeterFS'
dependencies = []
files = ['48838']
hgrepos = []
issue_num = 39327
keywords = ['patch']
message_count = 5.0
messages = ['359961', '360028', '408174', '413595', '413636', '413637']
nosy_count = 5.0
nosy_names = ['python-dev', 'serhiy.storchaka', 'miss-islington', 'PeterFS', 'lkantola']
pr_nums = ['31384', '31450', '31451']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39327'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']
    @PeterFS
    Copy link
    Mannequin Author

    PeterFS mannequin commented Jan 14, 2020 

    Python 3.6.9
Ubuntu 18.04
python3 -c 'import shutil; shutil.rmtree("1a")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 428, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 426, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 26] Text file busy: '4a'

    Reproduction method
    mkdir synched_folder\1a\2a\3a\4a
    mkdir synched_folder\1a\2b\3a\4a
    mkdir synched_folder\1b\2a\3a\4a
    choco install vagrant
    Save Vagrantfile in empty folder
    vagrant box add ubuntu/bionic64
    vagrant up
    vagrant ssh
    cd synched_folder
    python3 -c 'import shutil; shutil.rmtree("1a")'
    *** Error message ***
    rm -r 1a
    *** Works fine ***

    @PeterFS PeterFS mannequin added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error labels Jan 14, 2020
    @PeterFS
    Copy link
    Mannequin Author

    PeterFS mannequin commented Jan 15, 2020

    Problem is also reported in virtualbox
    https://www.virtualbox.org/ticket/19004

    From that ticket some more analysis is done;
    strace reveals that Python has kept an open fd for the directory being removed.

    @lkantola
    Copy link
    Mannequin

    lkantola mannequin commented Dec 10, 2021

    I have encountered the same problem with vboxsf mounted folders when using Debian guest on VirtualBox 6.1 hosted by Windows 10.

    The general VirtualBox issue of vboxsf not supporting rmdir when there are open file descriptors is difficult to solve, and out of scope for Python.

    However, it looks like the shutil.rmtree implementation can be easily modified to change the order of operations so that the rmdir is performed after the directory file descriptor is closed, which makes shutil.rmtree work under vboxsf shared folders.

    Example of the change against Python 3.7 is inlined below. The changed order matches the behavior of "rm -r" and to my understanding should not change the security implications of the code, but as the code is related to the prevention of a symlink attack https://bugs.python.org/issue4489 careful review would be needed.

    --- /usr/lib/python3.7/shutil.py--orig  2021-01-23 05:04:44.000000000 +0900
+++ /usr/lib/python3.7/shutil.py        2021-12-09 20:42:36.795338346 +0900
@@ -427,10 +427,6 @@
                 try:
                     if os.path.samestat(orig_st, os.fstat(dirfd)):
                         _rmtree_safe_fd(dirfd, fullname, onerror)
-                        try:
-                            os.rmdir(entry.name, dir_fd=topfd)
-                        except OSError:
-                            onerror(os.rmdir, fullname, sys.exc_info())
                     else:
                         try:
                             # This can only happen if someone replaces
@@ -442,6 +438,10 @@
                             onerror(os.path.islink, fullname, sys.exc_info())
                 finally:
                     os.close(dirfd)
+                try:
+                    os.rmdir(entry.name, dir_fd=topfd)
+                except OSError:
+                    onerror(os.rmdir, fullname, sys.exc_info())
         else:
             try:
                 os.unlink(entry.name, dir_fd=topfd)
@@ -489,10 +489,6 @@
         try:
             if os.path.samestat(orig_st, os.fstat(fd)):
                 _rmtree_safe_fd(fd, path, onerror)
-                try:
-                    os.rmdir(path)
-                except OSError:
-                    onerror(os.rmdir, path, sys.exc_info())
             else:
                 try:
                     # symlinks to directories are forbidden, see bug python/cpython#46010
@@ -501,6 +497,10 @@
                     onerror(os.path.islink, path, sys.exc_info())
         finally:
             os.close(fd)
+        try:
+            os.rmdir(path)
+        except OSError:
+            onerror(os.rmdir, path, sys.exc_info())
     else:
         try:
             if os.path.islink(path):

    @serhiy-storchaka
    Copy link
    Member

    New changeset b77158b by Lital Natan in branch 'main':
    bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
    b77158b

    @miss-islington
    Copy link
    Contributor

    New changeset 959846b by Miss Islington (bot) in branch '3.9':
    bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
    959846b

    @miss-islington
    Copy link
    Contributor

    New changeset 95d6271 by Miss Islington (bot) in branch '3.10':
    bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
    95d6271

    @serhiy-storchaka serhiy-storchaka added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Feb 21, 2022
    @serhiy-storchaka serhiy-storchaka added 3.9 only security fixes 3.10 only security fixes 3.11 only security fixes labels Feb 21, 2022
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.9 only security fixes 3.10 only security fixes 3.11 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    2 participants
    @serhiy-storchaka @miss-islington