classification
Title: Docs: the difference between rename and replace is not obvious
Type: behavior Stage:
Components: Documentation Versions: Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: asvetlov, docs@python, eryksun, pitrou, r.david.murray
Priority: low Keywords: easy

Created on 2016-08-28 19:07 by asvetlov, last changed 2016-08-28 22:56 by eryksun.

Messages (5)
msg273836 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2016-08-28 19:07
Hi.

On reading the doc for pathlib I've stuck with `.rename()` and `.replace()` (https://docs.python.org/3/library/pathlib.html#pathlib.Path.rename).

What's the difference?

Going to pathlib's source code I've figured out that methods are use different functions from `os` module: `os.rename()` and `os.replace()`.

But the documentation for `os` module is not obvious too: 
the docs for both functions are almost equal from my perspective, the only significant difference is that `os.rename()` suggests to use `os.replace()` for cross-compatibility.

Could anybody explain the difference?

Also, at least the doc for `pathlib.Path.rename` worth to have a sentence like borrowed from `os.rename`: "If you want cross-platform overwriting of the destination, use replace()."
msg273839 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-28 19:38
The existing docs are pretty clear on the difference: rename is only guaranteed to replace an existing file on unix (which I think means posix in this context), whereas replace always replaces the file, regardless of platform.

I'm actually surprised that rename is even part of the pathlib API, since its cross platform behavior is not consistent.  It also seems as though the replace docs are incorrect, since they imply directories are replaced, but the os docs say they are not.

The "this is a posix requirement" note in os.replace also seems imprecise.  Windows is not posix, so that note leaves me wondering if replace is atomic on Windows or not.  That should be clarified one way or another.

I think adding the cross platform note in the pathlib docs is reasonable, since it addresses the question of why there are two similar functions.
msg273840 - (view) Author: Andrew Svetlov (asvetlov) * (Python committer) Date: 2016-08-28 20:03
Just keeping the reference to `os.rename()`: https://docs.python.org/3/library/os.html#os.rename

Aha, `os.rename` says: "On Unix, if dst exists and is a file, it will be replaced silently if the user has permission." and "On Windows, if dst already exists, OSError will be raised even if it is a file."

Sorry, I've missed this.

Maybe it worth to add `.. warning:` section for both `os.rename` and `pathlib.Path.rename`?
msg273841 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-28 20:05
No, we don't like to overuse warnings.
msg273845 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2016-08-28 22:56
Having rename() in pathlib is fine as long as it links to the os docs. This probably needs a new issue, but I do see room for improvement in the latter.

For Unix, the os.rename and os.replace docs should clarify that an empty destination directory can be replaced by another directory. For example:

    >>> os.mkdir('foo1')
    >>> os.mkdir('foo2')
    >>> os.rename('foo1', 'foo2')

The specification of rename [1] states that "[i]f the *old* argument points to the pathname of a directory, the *new* argument shall not point to the pathname of a file that is not a directory". It further specifies that "[i]f *new* names an existing directory, it shall be required to be an empty directory". Windows, on the other hand, doesn't allow replacing a directory, even an empty one. The wording for os.rename and os.replace could be changed to something like: "[i]f dst is a directory, OSError will be raised, except not on Unix if src is a directory and dst is empty".

Windows MoveFileEx calls NtSetInformationFile to set the FileRenameInformation [2]. The MSDN docs do not explicitly require that the operation is atomic, unlike POSIX rename. I think it's a reasonable expectation that a Windows filesystem should rename atomically, and that's probably the case. However, it should be clear that only Unix guarantees this, e.g. "[i]f successful, on Unix the renaming will be an atomic operation, as required by POSIX".

MSDN claims that a rename will fail in the following cases:

    A file or directory can only be renamed within a volume. 

    Even if ReplaceIfExists is set to TRUE, the rename operation
    will still fail if a file with the same name already exists
    and is a directory, a read-only file, or a currently
    executing file.

    A file cannot be renamed if it has any open handles, unless it
    is only open because of a batch opportunistic lock (oplock)
    and the batch oplock can be broken immediately.

    A file cannot be renamed if a file with the same name exists
    and has open handles (except in the batch-oplock case
    described earlier).

    A directory cannot be renamed if it or any of its
    subdirectories contains a file that has open handles (except
    in the batch-oplock case described earlier).

For the third case, actually an open file can be renamed if delete/rename access is shared. For example:

    >>> with open('foo1', 'w') as f: f.write('foo1')
    ...
    4

Open the file with shared delete access:

    >>> SHARE_ALL = 7
    >>> _winapi.CreateFile('foo1', 0x80000000, SHARE_ALL, 0, 3, 0, 0)
    224

and rename/replace succeeds: 

    >>> os.replace('foo1', 'foo2')
    >>> open('foo2').read()
    'foo1'

Anyway, the os.replace docs could state in general that: "[o]n Windows, a PermissionError will be raised if dst is a read-only file, or if either src or dst is currently open, or if src is a directory with an open file".

[1]: http://pubs.opengroup.org/onlinepubs/9699919799/functions/rename.html
[2]: https://msdn.microsoft.com/en-us/library/ff540344
History
Date User Action Args
2016-08-28 22:56:53eryksunsetnosy: + eryksun
messages: + msg273845
2016-08-28 20:05:48r.david.murraysetmessages: + msg273841
2016-08-28 20:03:34asvetlovsetmessages: + msg273840
2016-08-28 19:38:47r.david.murraysetversions: - Python 3.4
2016-08-28 19:38:35r.david.murraysetnosy: + r.david.murray
messages: + msg273839
2016-08-28 19:07:12asvetlovcreate