classification
Title: [easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems
Type: enhancement Stage: needs patch
Components: Documentation, Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, eryksun, paul.moore, scienidlex, stephan, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: easy, newcomer friendly

Created on 2016-10-04 13:42 by stephan, last changed 2021-03-09 01:38 by scienidlex.

Messages (12)
msg278035 - (view) Author: stephan (stephan) Date: 2016-10-04 13:42
Hi,

I am just migrating my code from python 2.7.12 to 3.5.2
on Windows and stumbled on the following difference:

In python 2.7.12:
  os.rename(filepath1, filepath2) 
  works even if 
  filepath1 and filepath2 are on different drives
  (or one on a local drive, the other on a network share).

In python 3.5.2 I get for the same operation:
 "OSError: [WinError 17] The system cannot move the file to a different disk drive"

My question:
 - is this a bug?
 - if not, where is this difference mentioned in the docs?
   I did find nothing, but I think it should be mentioned,
   otherwise I assume it's a bug.
msg278036 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-04 13:50
On Python 3 on Windows, os.rename() is implemented as MoveFileExW() with flags=0.

The doc says: "When moving a file, the destination can be on a different file system or volume. If the destination is on another drive, you must set the MOVEFILE_COPY_ALLOWED flag in dwFlags."

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx?f=255&MSPPError=-2147217396

I guess that the portable fix is to try rename() or fall back on copy(src, dst) + delete(src).

--

On Python 2 on Windows, os.rename() is implemented as MoveFileW(). It seems like this function behaves as MoveFileEx() called with MOVEFILE_COPY_ALLOWED:

"A new file may be on a different file system or drive."
https://msdn.microsoft.com/en-us/library/windows/desktop/aa365239(v=vs.85).aspx

--

Should we add a flag to os.rename() to allow copy, to have a portable API?
msg278038 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-04 13:52
Ah, on Linux rename() also fails if src and dst are on two different filesystems: OSError: [Errno 18] Invalid cross-device link.

By the way, the Python document doesn't say anything about operation on two different filesystems :-/
https://docs.python.org/dev/library/os.html#os.rename
msg278040 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-10-04 14:00
3.3 added os.replace, which on Windows entailed a switch from calling MoveFile to MoveFileEx in order to specify the MOVEFILE_REPLACE_EXISTING flag. However, not passing the MOVEFILE_COPY_ALLOWED broke compatibility with os.rename on Windows for versions prior to 3.3. I don't know whether or not this was discussed as an intentional breaking change in order to align the behavior with POSIX rename(). The change seems reasonable to me, plus at this point I don't think much can be done other than to add a note to the docs that the behavior changed in 3.3.
msg278041 - (view) Author: stephan (stephan) Date: 2016-10-04 14:15
Hi,

I tryed os.replace() as replacement for os.rename() too,
but as you said it does not work if the files are on different drives.

For now I switched to shutil.move() but I suppose its not
as performant/optimal as an "move" or "rename" directly supported
by the OS.
msg278048 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-10-04 15:26
In scanning over issue 8828, I see no discussion of the consequences of not using MOVEFILE_COPY_ALLOWED in Antoine's patch, so it appears that this behavior change was unintentional.

> For now I switched to shutil.move() but I suppose its not
> as performant/optimal as an "move" or "rename" directly 
> supported by the OS.

Correct, the copy employed by shutil.move for a cross-volume move is not as optimized as what MoveFile uses (basically CopyFile), but unless you're moving a file that's very large it shouldn't matter, and even then without testing I don't know if it's a significant difference relative to the throughput of the disk(s) involved.
msg278113 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2016-10-05 07:34
> In scanning over issue 8828, I see no discussion of the consequences of not using MOVEFILE_COPY_ALLOWED in Antoine's patch, so it appears that this behavior change was unintentional.

Depending on your point of view, it *can* be seen as an enhancement.

See the documentation of MOVEFILE_COPY_ALLOWED: "If the file is
successfully copied to a different volume and the original file is
unable to be deleted, the function succeeds leaving the source file
intact." This behaviour can be seen as a source of bug. It can be
acceptable if if is expected.

Since shutil.move() already implements the copy+delete fallback, I
suggest to *not* change os.rename() but *document* the behaviour
change:

* os.rename() can fail if source and destination are on two different
file systems
* Use shutil.move() to support move to a different directory

https://docs.python.org/dev/library/shutil.html#shutil.move already
explains well the behaviour when two different filesystems are used:
"... os.rename() is used. Otherwise, src is copied to dst using
copy_function and then remove ..."

I also suggest to mention in shutil.move() doc that the source can be
left undeleted if delete fails for some reason when copy+delete is
used.
msg387667 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-02-25 12:46
The documentation of os.rename() should mention that on Windows the "operation will fail if src and dst are on different filesystems". For POSIX, instead of "will" it says "may", but the failure is a certainty in Windows since MOVEFILE_COPY_ALLOWED isn't used.
msg388283 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2021-03-08 18:57
This issue is now a matter of *documenting* the Windows behavior. Is there any volunteer to propose a PR?
msg388295 - (view) Author: T-VEy (scienidlex) Date: 2021-03-08 20:06
i have problem with python version 3.9.1
msg388297 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-08 20:18
T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.
msg388322 - (view) Author: T-VEy (scienidlex) Date: 2021-03-09 01:38
Eryk, can you please go on details on how are they using different strategies to fix bugs but still it's not enough for the time being?Because I know nothing about python and I recognized it in a minute of opening the files.I don't even have it installed on my pc.I don't even have python, not even version 2.7. 

  On Mon, Mar 8, 2021 at 23:48, Eryk Sun<report@bugs.python.org> wrote:   
Eryk Sun <eryksun@gmail.com> added the comment:

T-VEy, this is a documentation-only issue for versions of Python that are under active development for bug fixes and enhancements -- 3.8, 3.9, and 3.10.

----------
versions: +Python 3.10, Python 3.8

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue28356>
_______________________________________
History
Date User Action Args
2021-03-09 01:38:28scienidlexsetmessages: + msg388322
2021-03-08 20:18:47eryksunsetmessages: + msg388297
versions: + Python 3.8, Python 3.10
2021-03-08 20:06:59scienidlexsetnosy: + scienidlex

messages: + msg388295
versions: - Python 3.8, Python 3.10
2021-03-08 18:57:11vstinnersetmessages: + msg388283
2021-03-08 18:56:42vstinnersetkeywords: + easy, newcomer friendly
title: Windows: os.rename different in python 2.7.12 and python 3.5.2 -> [easy doc] Document os.rename() behavior on Windows when src and dst are on different filesystems
2021-03-05 12:26:34eryksunsettype: behavior -> enhancement
versions: - Python 3.5, Python 3.6, Python 3.7
2021-02-25 12:46:58eryksunsetassignee: docs@python
components: + Documentation
versions: + Python 3.6, Python 3.7, Python 3.8, Python 3.9, Python 3.10, - Python 2.7
nosy: + docs@python

messages: + msg387667
stage: needs patch
2016-10-05 07:34:03vstinnersetmessages: + msg278113
2016-10-04 15:26:35eryksunsetmessages: + msg278048
2016-10-04 14:15:02stephansetmessages: + msg278041
2016-10-04 14:00:10eryksunsetnosy: + eryksun
messages: + msg278040
2016-10-04 13:52:44vstinnersetmessages: + msg278038
2016-10-04 13:50:31vstinnersettitle: os.rename different in python 2.7.12 and python 3.5.2 -> Windows: os.rename different in python 2.7.12 and python 3.5.2
nosy: + paul.moore, tim.golden, vstinner, zach.ware, steve.dower

messages: + msg278036

components: + Windows
2016-10-04 13:42:32stephancreate