This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: os.replace is not cross-platform: at least improve documentation
Type: behavior Stage: resolved
Components: Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eric.smith, eryksun, hancos
Priority: normal Keywords:

Created on 2021-12-06 21:20 by hancos, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (6)
msg407864 - (view) Author: nobody (hancos) Date: 2021-12-06 21:20
When using threads, os.replace gives no errors on Linux. On Windows however you get PermissionErrors like: '[WinError 5] Access is denied' for os.replace and '[Errno 13] Permission denied' when reading the os.replace file.
The only way I could get this to work was to add retries with delay.
At least the documentation should be improved.
msg407876 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-12-06 23:14
I suspect anti-virus software.
msg407884 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-12-07 00:47
The os module tries to avoid documenting low-level OS behaviors. It would be unreliable and difficult to maintain. 

In the case of os.replace(), in Windows it calls MoveFileExW() with the flag MOVEFILE_REPLACE_EXISTING [1]. The source and destination paths must be on the same volume. The caller must have permission to delete the source path and, if it already exists, the destination path. The caller must have permission to add a file or directory to the destination directory. Existing opens of the source path must share delete access. If the source path is a directory, none of the files and directories in its tree is allowed to be open. If the destination path already exists, it cannot be a directory, a readonly file, or mapped as a process image (i.e. a data mapping is allowed, but an executing EXE or DLL is disallowed). Currently, existing opens of the destination path are not allowed, even if they share delete access. 

In Windows 10+, there's new support at the NT system call level (i.e. the layer beneath the Windows API) to support replacing a file that has existing opens, if the file system supports it (e.g. NTFS, ReFS). Delete access is still required, so the opens must share delete access. MoveFileExW() has not been updated yet to use this new capability. If and when it's supported, the requirement for existing opens to share delete access means it won't help in general. Most Windows programs, including Python, do not share delete access on open files. To share delete access in Python, one can use a custom opener function that calls CreateFileW() and wraps the OS handle in an fd via msvcrt.open_osfhandle().

---
[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw
msg407961 - (view) Author: nobody (hancos) Date: 2021-12-07 18:46
Thank you for your replies. I have been reading more and conclude that I/O programming on Windows and Linux are two different things.

Python is not cross-platform in the sense that a Python program always works on different operating systems but that you can make it work on different operating systems (most of the time).

My application uses the PyPi package cachelib. The file system interface is implemented in this file:

https://github.com/pallets/cachelib/blob/main/src/cachelib/file.py

This package uses os.replace and other file I/O methods. 

From what I understand is that this package must be modified to also handle WinError 5 and Errno 13 if you want to use it on Windows,
meaning more specific exception handling and adding retries and a delay for every I/O related function accessing the os.replace file.

Please correct me if I am wrong.
msg407974 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2021-12-07 21:51
As they say, there's no such thing as "portable software", only "software that has been ported". Especially in an area like file I/O: once you move beyond simple "one process opens, writes, and closes; and another process then opens, reads, and closes", there are a lot of platform-specific issues. Python does not try to abstract away all possible file I/O issues.
msg408003 - (view) Author: nobody (hancos) Date: 2021-12-08 09:08
Thank you again, closing this issue.
History
Date User Action Args
2022-04-11 14:59:53adminsetgithub: 90161
2021-12-08 09:46:58eric.smithsetresolution: not a bug
2021-12-08 09:08:26hancossetstatus: open -> closed

messages: + msg408003
stage: resolved
2021-12-07 21:51:42eric.smithsetmessages: + msg407974
2021-12-07 18:46:11hancossetmessages: + msg407961
2021-12-07 00:47:39eryksunsetnosy: + eryksun
messages: + msg407884
2021-12-06 23:14:51eric.smithsetnosy: + eric.smith
messages: + msg407876
2021-12-06 21:20:05hancoscreate