This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: shutil.copy() inefficient implementation in Windows
Type: performance Stage:
Components: Windows Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, sfmc, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2021-07-07 12:11 by sfmc, last changed 2022-04-11 14:59 by admin.

Messages (3)
msg397076 - (view) Author: sfmc (sfmc) Date: 2021-07-07 12:11
In Windows shutil.copy() uses _copyfileobj_readinto which copies file in user mode.
In Windows there is an fast API to copy file in kernel mode: CopyFile (see https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfile).
msg397077 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-07-07 12:53
> In Windows there is an fast API to copy file in kernel mode: CopyFile

The possibility of calling CopyFileEx() for shutil.copy2() is discussed in issue 30044. Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy. It opens the source and destination files and makes multiple system calls in order to copy file data and metadata (e.g. system calls such as NtOpenFile, NtCreateFile, NtReadFile, NtWriteFile, NtQueryInformationFile, NtSetInformationFile, NtQueryEaFile, NtSetEaFile, NtQuerySecurityObject, etc). This includes copying the primary data stream, alternate data streams, file attributes, extended file attributes, and security resource attributes.
msg397091 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2021-07-07 14:32
> Note that CopyFileEx() is a high-level Windows API function, not a "kernel mode" copy.

This is true today, but could change whenever Windows feels like changing it. If we switch to the native API then we'll get any advantage there automatically.

The only challenge is in managing changes to semantics (that is, anything extra we do to "match" Unix that isn't normally how copies work on Windows - personally, I'd rather be more native on Windows anyway).
History
Date User Action Args
2022-04-11 14:59:47adminsetgithub: 88745
2021-07-07 14:39:41eryksunsetcomponents: + Windows, - Library (Lib)
2021-07-07 14:32:41steve.dowersetmessages: + msg397091
2021-07-07 12:53:57eryksunsetnosy: + eryksun
messages: + msg397077
components: - Windows
2021-07-07 12:31:11xtreaksetnosy: + paul.moore, tim.golden, zach.ware, steve.dower
components: + Windows
2021-07-07 12:11:29sfmccreate