classification
Title: shutil.copystat should (allow to) copy ownership, and other attributes
Type: enhancement Stage: needs patch
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, eryksun, giampaolo.rodola, noctiflore
Priority: normal Keywords: easy

Created on 2017-04-11 15:42 by noctiflore, last changed 2020-07-21 01:10 by eryksun.

Messages (7)
msg291499 - (view) Author: Florent Coriat (noctiflore) Date: 2017-04-11 15:42
shutil.copystat() copies permissions, timestamps and even flags and xattrs (if supported), but not ownership.
Furthermore, shutil.copy2() documentation until 2.7 used to say it behaves like cp -p, which preserves ownership, and not xattr nor flags. (On my system it silently fails to copy ownership when not root).

It may not be related, but comments in source code for the except NotImplementedError block concerning chmod mistakenly mentions chown-related functions.

I think copystat (and copy2) should at least provide an option to preserve ownership.
I do not know if it currently preserves SELinux context and ACL, but if not, it may also allow it.
It would be really useful for replication or backup applications to have a function that copies everything it can.
msg319373 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-06-12 10:09
This could be achieved at least on Windows with CopyFileEx [1] and on OSX with copyfile(3) + COPYFILE_ALL which copies ACLs (but not users/groups). These are (were, in case of CopyFileEx) exposed in https://github.com/python/cpython/pull/7160/. Such a new functionality would probably deserve a separate copy3() function but it would serve OSX and Windows only. I'm not sure how things would work on other POSIX platforms. 

[1] https://msdn.microsoft.com/en-us/library/windows/desktop/aa363852(v=vs.85).aspx
[2] http://www.manpagez.com/man/3/copyfile/
msg373445 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2020-07-10 07:38
POSIX ACLs and SELinux context information are stored in extended file attributes. The information is copied from source to destination. POSIX ACLs are stored in xattr "system.posix_acl_access" and SELinux context in xattr "security.selinux".
msg373471 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-07-10 16:44
In Windows, I wouldn't expect shutil.copy2 to preserve the owner and ACLs. They change whenever a file gets copied via CopyFileExW [1]. Keeping them exactly as in the source file generally requires a privileged backup and restore operation, such as via BackupRead [2] and BackupWrite [3]. Unless the caller has SeRestorePrivilege, the owner can only be set to one of the SIDs in the caller's groups that are flagged as SE_GROUP_OWNER, which is usually just the user's SID or, for an admin, the Administrators SID. Also, for copying the system ACL, adding or removing audit and scoped-policy-identifier entries requires SeSecurityPrivilege.

CopyFileExW copies all data streams in a file, which is typically just the anonymous data stream, but an NTFS/ReFS file can have multiple named data streams. For metadata, it copies the change and modify timestamps (but not the create and access timestamps), file attributes (readonly, hidden, system, archive, temporary, not-content-indexed), extended attributes, and resource attributes [4]. 

Separating this functionality into shutil.copy and shutil.copystat would be fairly involved. These functions could be left as is and just document the discrepancy in shutil.copy2, or new functions could be implemented in the nt or _winapi module to list the data streams in a file and get/set file attributes and system resource attributes. Supporting extended attributes would require the native NT API, and for little benefit since they're mostly used for "$Kernel." prefixed attributes that can only be set by kernel-mode callers such as drivers.

---

[1]: https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexw
[2]: https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-backupread
[3]: https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-backupwrite
[4]: Resource attributes are like extended attributes, but a named resource attribute is a tuple of one or more items with a given data type (integer, string, or bytes) that's stored as an entry in the file's system ACL. Keeping them in the SACL allows conditional access/audit entries to reference them in an access check or access audit. Unlike audit entries in the SACL, reading and writing resource attributes doesn't require SeSecurityPrivilege.
msg373998 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2020-07-20 11:01
Since the need to copy file ownership is common, I think there could be space for a new copy3() function which copies ownership + extended attributes (where possible). In detail:

- on Windows this can be achieved by using CopyFileEx, which would also provide better overall performances than the current readinto() implementation. This was the original approach used in BPO-33671, discarded after Eryk pointed out that ACLs should not be copied.

- on macOS we can use fcopyfile() (which is already exposed privately) + COPYFILE_ALL, see http://www.manpagez.com/man/3/fcopyfile/

- on all other UNIX we can use the current shutil.copystat() (which already copies xattrs) + os.chown(). I am not entirely sure this would achieve a "full ACLs copy" though, or what would be needed exactly to achieve that.
msg373999 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2020-07-20 11:37
Sorry, after re-reading Eryk's comment, it seems I'm not correct about CopyFileEx.
msg374044 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-07-21 01:10
> Since the need to copy file ownership is common, I think there could
> be space for a new copy3() function which copies ownership + extended
> attributes (where possible).

FYI, Windows and POSIX have significantly different concepts about file (object) ownership. In Windows:

    * Any type of SID can be set as the owner, such as a user, global
      group, local group, well-known group, domain, or logon session. All
      of these SID types, except for user SIDs, are commonly set in the
      groups of a token. Also, the token user is not limited to just users.
      It's commonly set to a well-known group such as SYSTEM, LOCAL
      SERVICE, or NETWORK SERVICE.

    * The effective access token of a thread is granted owner rights to
      an object if the token user or any of the token's enabled groups is
      the owner of the object. For example, if an object is owned by the
      "BUILTIN\Users" local group, then all access tokens for standard-user
      logons will be granted owner rights as long as they have the
      "BUILTIN\Users" group enabled, which it is by default.
      
    * If not set explicitly via "OWNER RIGHTS" (i.e. S-1-3-4), the 
      owner is implicitly granted the READ_CONTROL right to query the
      object security and the WRITE_DAC right to modify the object's
      resource attributes and discretionary access-control list. As 
      long as these rights are granted implicitly, they cannot be
      denied by deny access-control entries. However, implicit owner
      rights may be denied if an object has an implicit (by object 
      type) or explicit (by label) no-read-up or no-write-up mandatory
      policy, and the token's integrity level is less than that of the
      object.

    * An explicit "OWNER RIGHTS" entry can be set in the discretionary
      access control list in order to override the implicit owner rights.
      This is not the same as setting owner rights in POSIX, since other
      ACL entries may grant or deny rights. Given the canonical priority 
      of deny access-control entries and also mandatory access control
      based on the integrity level of the object vs the token, granting
      explicit access to "OWNER RIGHTS" does not necessarily ensure the
      owner will even be granted at least the desired access in all
      contexts. Also, unlike the implicit case, if an "OWNER RIGHTS"
      entry grants READ_CONTROL and/or WRITE_DAC access, either right 
      may be denied by deny access-control entries.
History
Date User Action Args
2020-07-21 01:10:15eryksunsetmessages: + msg374044
2020-07-20 11:37:45giampaolo.rodolasetmessages: + msg373999
2020-07-20 11:01:51giampaolo.rodolasetmessages: + msg373998
2020-07-10 16:44:52eryksunsetnosy: + eryksun
messages: + msg373471
2020-07-10 07:38:06christian.heimessetversions: + Python 3.10, - Python 3.7
nosy: + christian.heimes

messages: + msg373445

keywords: + easy
stage: needs patch
2018-06-12 10:09:43giampaolo.rodolasetnosy: + giampaolo.rodola
messages: + msg319373
2017-04-14 17:59:26serhiy.storchakasetversions: - Python 3.5, Python 3.6
2017-04-11 15:42:33noctiflorecreate