This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: On Windows, shutil.move doesn't raise FileExistsError if dst exists like os.rename
Type: behavior Stage:
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, fireattack, paul.moore, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-01-14 21:29 by fireattack, last changed 2022-04-11 14:59 by admin.

Messages (5)
msg385083 - (view) Author: fireattack (fireattack) * Date: 2021-01-14 21:29
According to https://docs.python.org/3.8/library/shutil.html#shutil.move

"If the destination already exists but is not a directory, it may be overwritten depending on os.rename() semantics."

I interpret "depending on os.rename() semantics" to mean it will follow os.rename()'s behavior.

According to https://docs.python.org/3/library/os.html#os.rename

"On Windows, if dst exists a FileExistsError is always raised."

However, their behaviors are not the same.

For os.rename, it does raise FileExistsError if dst exists.
For shutil.move, it silently overwrites dst.

It's either a bug in behavior of shutil.move, or the documentation need to be updated.
msg385084 - (view) Author: fireattack (fireattack) * Date: 2021-01-14 21:29
Sorry, I should link https://docs.python.org/3/library/shutil.html#shutil.move for latest doc. But the content is the same.
msg385090 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-14 23:23
shutil.move() has always fallen back on copy/unlink if os.rename() fails. This used to be clearly documented to account for the behavior of os.rename() in Windows, but the details were obscured over multiple rewrites. Currently the documentation makes it seem that copy/unlink are only used if the destination is on another filesystem.

Anyway, falling back on copy/unlink is sub-optimal as a move operation [1] and should be avoided wherever possible. It would be better to use os.replace() instead of os.rename(). This expands support for an atomic move within a filesystem, and it avoids the need to modify the documentation, except to reference os.replace() instead of os.rename().

---

[1] As implemented, a copy will not retain the source file's alternate data streams, security descriptor (i.e. owner and access/audit control), file attributes, or extended file attributes. There are additional problems with reparse points other than symlinks, but that's a difficult problem. We don't support raw copying of reparse points, or even high-level support for creating mountpoint (aka junction) reparse points. There's _winapi.CreateJunction, but the current implementation is just for testing purposes.
msg385107 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-01-15 11:10
Eryk Sun, do you mind to create a PR?
msg385116 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-01-15 16:18
I can help, but in this case there isn't much to do. Just replace os.rename() with os.replace(), make a minor doc change, and maybe add a test that ensures a junction can be moved over an existing file on the same filesystem. For example:

    >>> os.mkdir('temp')
    >>> _winapi.CreateJunction('temp', 'src')
    >>> os.lstat('src').st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT
    True
    >>> open('dst', 'w').close()

The current implementation tries copytree() on the junction mountpoint and fails to create a new directory named "dst":

    >>> try: shutil.move('src', 'dst')
    ... except FileExistsError as e: e
    ...
    FileExistsError(17, 'Cannot create a file when that file already exists')

But move() should simply replace "dst" with the junction via os.replace():

    >>> os.replace('src', 'dst')
    >>> os.lstat('dst').st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT
    True
History
Date User Action Args
2022-04-11 14:59:40adminsetgithub: 87095
2021-01-15 16:18:14eryksunsetmessages: + msg385116
2021-01-15 11:10:48serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg385107
2021-01-14 23:23:47eryksunsetnosy: + paul.moore, tim.golden, zach.ware, steve.dower

components: + Windows
versions: + Python 3.10
2021-01-14 23:23:32eryksunsetnosy: + eryksun
messages: + msg385090
2021-01-14 21:29:56fireattacksetmessages: + msg385084
2021-01-14 21:29:10fireattackcreate