This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: os.setxattr PermissionError on panfs propagates up causing `copystat`, `copytree`, and `pip install .` to fail unhepfully
Type: crash Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Bart Oldeman, Gerrit.Holl, Maxime Boissonneault, giampaolo.rodola, iritkatriel, obilaniu, r.david.murray, tarek
Priority: normal Keywords: patch

Created on 2015-06-30 12:07 by Gerrit.Holl, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 13212 merged python-dev, 2019-05-09 00:18
PR 13234 merged miss-islington, 2019-05-10 03:22
Messages (10)
msg245985 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2015-06-30 12:07
`shutil.copystat` fails on [panfs](https://en.wikipedia.org/wiki/Panasas#PanFS) if the source file lacks u+w, because setting extended attributes results in a `PermissionError`.  This leads to higher end functions such as `shutil.copytree` to fail.  More seriously, it leads to `pip install .` to fail, as I [reported here](https://github.com/pypa/pip/issues/2941):

    In [55]: shutil.copystat("/src/on/panfs", "/tmp/fubar")
    ---------------------------------------------------------------------------
    PermissionError                           Traceback (most recent call last)
    <ipython-input-55-139c9fc77184> in <module>()
    ----> 1 shutil.copystat("/home/users/gholl/checkouts/pyatmlab/.git/objects/pack/pack-c1449559ec4287b3830efe2913608dddf2f21391.pack", "/tmp/fubar")
    
    /home/users/gholl/lib/python3.4/shutil.py in copystat(src, dst, follow_symlinks)
        211             else:
        212                 raise
    --> 213     _copyxattr(src, dst, follow_symlinks=follow)
        214
        215 def copy(src, dst, *, follow_symlinks=True):
    
    /home/users/gholl/lib/python3.4/shutil.py in _copyxattr(src, dst, follow_symlinks)
        151             try:
        152                 value = os.getxattr(src, name, follow_symlinks=follow_symlinks)
    --> 153                 os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
        154             except OSError as e:
        155                 if e.errno not in (errno.EPERM, errno.ENOTSUP, errno.ENODATA):
    
    PermissionError: [Errno 13] Permission denied: '/tmp/fubar'

This occurs for any source file where the user write bit is not set.

Now, I'm not sure if this should be addressed in `pip`, `shutil.copytree`, `shutil.copystat`, `setxattr`, `panfs`, or in none of the above.  I'm reporting it in different places; please close this issue if this is the wrong place.
msg245987 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-06-30 13:08
There are a couple of related open issues.  I think there is an stdlib problem here, but I"m not sure what the solution is.
msg245989 - (view) Author: Gerrit Holl (Gerrit.Holl) * Date: 2015-06-30 13:26
Perhaps the solution would be some kind of flag, at least for copytree and possibly others, on what to do when attributes cannot be completely copied — a bit like numpys options to raise, warn, or ignore?
msg301493 - (view) Author: Bart Oldeman (Bart Oldeman) Date: 2017-09-06 17:50
The issue can be avoided by calling _copyxattr *before* instead of after os.chmod in shutil.copystat. That way the file is still writable.

The same issue happens on the Lustre parallel file system, with the lustre.lov extended attribute.
msg342022 - (view) Author: Olexa Bilaniuk (obilaniu) * Date: 2019-05-10 00:49
This old bug now has a PR on Github authored by myself that may be reviewed.
msg342051 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-10 03:20
Patch LGTM. I'd like to point out one thing for posterity. Current shutil code catches EPERM but not EACCES. While reviewing the patch I wondered whether simply catching (and ignoring) both error codes instead, but it turns out they are supposed to have 2 different meanings:
https://stackoverflow.com/a/35879961/376587
Especially after _copyxattr is moved before chmod() EACCES (permission denied) should propagate instead of being silenced, and EPERM should be left in place because in this case EPERM acts more like an ENOTSUP (operation not supported) than EACCES (permission denied) and as such should rightly be ignored.
msg342052 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-10 03:22
New changeset 79efbb719383386051c72f2ee932eeca8e033e6b by Giampaolo Rodola (Olexa Bilaniuk) in branch 'master':
bpo-24538: Fix bug in shutil involving the copying of xattrs to read-only files. (PR-13212)
https://github.com/python/cpython/commit/79efbb719383386051c72f2ee932eeca8e033e6b
msg342433 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-14 05:30
New changeset 0a5b88e7f23b671d63896619b13148b0e4e2b5dd by Giampaolo Rodola (Miss Islington (bot)) in branch '3.7':
bpo-24538: Fix bug in shutil involving the copying of xattrs to read-only files. (PR-13212) (#13234)
https://github.com/python/cpython/commit/0a5b88e7f23b671d63896619b13148b0e4e2b5dd
msg378292 - (view) Author: Irit Katriel (iritkatriel) * (Python committer) Date: 2020-10-08 23:14
This seems complete, can it be closed?
msg378300 - (view) Author: Olexa Bilaniuk (obilaniu) * Date: 2020-10-09 03:51
Yes, it may be closed. The fix was merged in time to make it into Python 3.7.4 and 3.8.0, and solved the problems we were facing with fancy filesystem xattrs on read-only files.
History
Date User Action Args
2022-04-11 14:58:18adminsetgithub: 68726
2020-10-20 10:57:24iritkatrielsetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2020-10-09 03:51:28obilaniusetmessages: + msg378300
2020-10-08 23:14:59iritkatrielsetnosy: + iritkatriel
messages: + msg378292
2019-05-14 05:30:28giampaolo.rodolasetmessages: + msg342433
2019-05-10 03:26:24giampaolo.rodolasetversions: + Python 3.7, Python 3.8, - Python 3.2, Python 3.3, Python 3.4
2019-05-10 03:22:47miss-islingtonsetpull_requests: + pull_request13144
2019-05-10 03:22:24giampaolo.rodolasetmessages: + msg342052
2019-05-10 03:20:02giampaolo.rodolasetmessages: + msg342051
2019-05-10 00:49:13obilaniusetnosy: + giampaolo.rodola, tarek, obilaniu
messages: + msg342022
2019-05-09 00:18:50python-devsetkeywords: + patch
stage: patch review
pull_requests: + pull_request13123
2017-09-06 17:54:38Maxime Boissonneaultsetnosy: + Maxime Boissonneault
2017-09-06 17:50:12Bart Oldemansetnosy: + Bart Oldeman
messages: + msg301493
2015-06-30 13:26:51Gerrit.Hollsetmessages: + msg245989
2015-06-30 13:08:49r.david.murraysetnosy: + r.david.murray
messages: + msg245987
2015-06-30 12:07:29Gerrit.Hollsettype: crash
2015-06-30 12:07:21Gerrit.Hollcreate