classification
Title: shutil.move fails to move symlink (Invalid cross-device link)
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.10, Python 3.9, Python 3.8, Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Jeffrey.Kintscher, Murray Wilson, Unode, tarek
Priority: normal Keywords: patch

Created on 2016-04-17 16:58 by Unode, last changed 2020-08-06 23:23 by Jeffrey.Kintscher.

Pull Requests
URL Status Linked Edit
PR 21759 open Jeffrey.Kintscher, 2020-08-06 23:23
Messages (5)
msg263616 - (view) Author: Renato Alves (Unode) Date: 2016-04-17 16:58
Hi everyone,

I'm not really sure if this is a new issue but digging through the bug reports from the past I couldn't find an answer.
There's http://bugs.python.org/issue1438480 but this seems to be a different issue.
I also found http://bugs.python.org/issue9993 that addressed problems with symlinks but didn't correct the behavior reported here.

The problem can be visualized with the following code.
Code fails on python 2.7 as well as python 3.4+. Not tested in python <2.7 and <3.4.


    import shutil
    import os
    
    TMPDIR = "/tmp/tmpdir"
    TESTLINK = "test_dir"
    
    if not os.path.isdir(TMPDIR):
        os.mkdir(TMPDIR)
    
    if not os.path.islink(TESTLINK):
        os.symlink(TMPDIR, TESTLINK)
    
    shutil.move(TESTLINK, TMPDIR)


When executed it gives me:

    % python3 test.py
    Traceback (most recent call last):
      File "test.py", line 14, in <module>
        shutil.move(TESTLINK, TMPDIR)
      File "/usr/lib64/python3.4/shutil.py", line 516, in move
        os.rename(src, dst)
    OSError: [Errno 18] Invalid cross-device link: 'test_dir' -> '/tmp/tmpdir'


This happens because /tmp is:

  tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime,nodiratime)


In the past the recommendation to handle this problem was to stop using os.rename and use shutil.move instead.
This was even discussed in a bug report - http://bugs.python.org/issue14848

If one searches for this exception there's plenty of advice [1][2][3][4] in the same direction.
However, given that shutil.move uses os.rename internally, the problem returns.

On the other end doing the equivalent action in the shell with 'mv' works fine.


[1] - http://stackoverflow.com/a/15300474
[2] - https://mail.python.org/pipermail/python-list/2005-February/342892.html
[3] - http://www.thecodingforums.com/threads/errno-18-invalid-cross-device-link-using-os-rename.341597/
[4] - https://github.com/pypa/pip/issues/103
msg263617 - (view) Author: Renato Alves (Unode) Date: 2016-04-17 17:01
Also related to http://bugs.python.org/issue212317
msg263654 - (view) Author: SilentGhost (SilentGhost) * (Python triager) Date: 2016-04-18 08:09
This seems to be only triggered when moving a symlink into its destination. Assumption in the code is that when _samefile returns True, it must be due to case-insensitive filesystem, rather than this edge case.
msg374648 - (view) Author: Murray Wilson (Murray Wilson) Date: 2020-07-31 17:22
It also happens when moving a file in linux from an xfs filesystem to a NFS mounted filesystem.
msg374916 - (view) Author: Jeffrey Kintscher (Jeffrey.Kintscher) * Date: 2020-08-06 07:10
SilentGhost's analysis is correct.  The provided example code causes the error because it is trying to move the symlink into its target when the target is a directory. Any cross-device moving issues are unrelated to this example code.  Here is the relevant code in the master branch:

    if os.path.isdir(dst):
        if _samefile(src, dst):
            # We might be on a case insensitive filesystem,
            # perform the rename anyway.
            os.rename(src, dst)
            return

shutil._samefile() considers the example link and its target to be the same.  When _samefile() returns False, this code gets executed:

        real_dst = os.path.join(dst, _basename(src))

        if os.path.exists(real_dst):
            raise Error("Destination path '%s' already exists" % real_dst)
    try:
        os.rename(src, real_dst)
    except OSError:
        if os.path.islink(src):
            linkto = os.readlink(src)
            os.symlink(linkto, real_dst)
            os.unlink(src)

A simple fix is to check whether src is a symlink when _samefile() returns True.  The "Destination path...already exists" error isn't a problem for our symlink case because the shell mv command also returns an error.

$ ls -l /tmp/tmpdir/
total 0
lrwxr-xr-x  1 jeff  staff  11 Aug  5 23:36 test_dir -> /tmp/tmpdir
$ mv test_dir /tmp/tmpdir
mv: test_dir and /tmp/tmpdir/test_dir are identical

I will generate a pull request.
History
Date User Action Args
2020-08-06 23:23:56Jeffrey.Kintschersetkeywords: + patch
stage: patch review
pull_requests: + pull_request20904
2020-08-06 07:27:56Jeffrey.Kintschersetversions: + Python 3.7, Python 3.8, Python 3.9, Python 3.10
2020-08-06 07:10:23Jeffrey.Kintschersetmessages: + msg374916
2020-07-31 18:02:55SilentGhostsetnosy: - SilentGhost
2020-07-31 17:22:31Murray Wilsonsetnosy: + Murray Wilson
messages: + msg374648
2019-06-01 20:46:10Jeffrey.Kintschersetnosy: + Jeffrey.Kintscher
2016-04-18 08:09:08SilentGhostsetversions: + Python 3.6, - Python 3.4
nosy: + SilentGhost, tarek

messages: + msg263654

type: behavior
2016-04-17 17:01:21Unodesetmessages: + msg263617
2016-04-17 16:59:06Unodesetversions: + Python 2.7, Python 3.5
2016-04-17 16:58:17Unodecreate