classification
Title: shutil.copytree fails when copying NFS to NFS
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Albert.Zeyer, Gabriel Devenyi, Michael Burt, cdyson37, cheryl.sabella, desbma, giampaolo.rodola, hans-meine, hynek, jhamrick, larry, minrk, r.david.murray
Priority: normal Keywords: patch

Created on 2015-07-04 20:05 by jhamrick, last changed 2019-09-09 21:58 by Michael Burt. This issue is now closed.

Files
File name Uploaded Description Edit
bug_demo.py jhamrick, 2015-07-04 20:05 Reproduces the copytree error on Python 3.4
Pull Requests
URL Status Linked Edit
PR 8601 closed giampaolo.rodola, 2018-08-01 07:43
PR 13369 merged yingw787, 2019-05-16 23:38
PR 13673 merged miss-islington, 2019-05-30 03:27
Messages (13)
msg246272 - (view) Author: Jess Hamrick (jhamrick) * Date: 2015-07-04 20:05
shutil.copytree seems to fail when copying files across NFS filesystems. In this example (see bug_demo.py), /tmp is a normal ext4 filesystem and the current working directory is NFS (version 4). Interestingly, it works fine to to copy between ext4 and NFS, just not NFS and NFS (even if it's the same NFS mount).

Depending on the specific version of 3.4, there are slightly different errors. For 3.4.0:

$ uname -a
Linux compmodels-node 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ python3 bug_demo.py

--------------------------------------------------
Python version info:
3.4.0 (default, Jun 19 2015, 14:20:21)
[GCC 4.8.2]
--------------------------------------------------

Copying NFS --> /tmp
Copying /tmp --> NFS
Copying NFS --> NFS
Traceback (most recent call last):
  File "/usr/lib/python3.4/shutil.py", line 336, in copytree
    copystat(src, dst)
  File "/usr/lib/python3.4/shutil.py", line 212, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/lib/python3.4/shutil.py", line 152, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 22] Invalid argument: 'demo_files3'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bug_demo.py", line 27, in <module>
    shutil.copytree('demo_files', 'demo_files3')
  File "/usr/lib/python3.4/shutil.py", line 339, in copytree
    if why.winerror is None:
AttributeError: 'OSError' object has no attribute 'winerror'


And for 3.4.3:

$ uname -a
Linux compmodels-node-02 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ python3 bug_demo.py

--------------------------------------------------
Python version info:
3.4.3 (default, Mar 26 2015, 22:03:40)
[GCC 4.9.2]
--------------------------------------------------

Copying NFS --> /tmp
Copying /tmp --> NFS
Copying NFS --> NFS
Traceback (most recent call last):
  File "bug_demo.py", line 27, in <module>
    shutil.copytree('demo_files', 'demo_files3')
  File "/usr/lib/python3.4/shutil.py", line 343, in copytree
    raise Error(errors)
shutil.Error: [('demo_files', 'demo_files3', "[Errno 22] Invalid argument: 'demo_files3'")]


This is probably related to https://bugs.python.org/issue17076 and my guess is that's also why there is a difference in the error messages between 3.4.0 and 3.4.3.
msg246273 - (view) Author: Jess Hamrick (jhamrick) * Date: 2015-07-04 20:26
Some further information: if I run copystat directly on 3.4.3, I get essentially the same error as on 3.4.0. So really it only looks like the difference is just in how the error is reported:


Traceback (most recent call last):
  File "bug_demo.py", line 31, in <module>
    shutil.copystat('demo_files', 'demo_files3')
  File "/usr/lib/python3.4/shutil.py", line 213, in copystat
    _copyxattr(src, dst, follow_symlinks=follow)
  File "/usr/lib/python3.4/shutil.py", line 153, in _copyxattr
    os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
OSError: [Errno 22] Invalid argument: 'demo_files3'


Also, I should mention that I did test this on 2.7 as well, and it is not an issue there.
msg246278 - (view) Author: Min RK (minrk) * Date: 2015-07-04 21:19
On a bit of further investigation, the NFS files have an xattr `system.nfs4_acl`. This can be read, but attempting to write it fails with EINVAL. Attempting to copy from NFS to non-NFS fails with ENOTSUP, which is caught and ignored, but copying from NFS to NFS raises EINVAL, which raises.

Adding `EINVAL` to the ignored errnos would fix the problem, but might hide real failures (I'm not sure about the real failures, but it seems logical).

Since the `copy_function` is customizable to switch between `copy` and `copy2`, making copystat optional on files, perhaps the `copystat` should be optional on directories, as well.
msg246279 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-07-04 21:45
There are a number of open issues with copytree originating from copystat.  It would be great if someone could pull them all together and propose a solution.  Making it optional might indeed be the best solution.
msg280051 - (view) Author: Gabriel Devenyi (Gabriel Devenyi) Date: 2016-11-04 15:16
I'm running into this issue with python 3.5 and a ZFS-backed NFS4 mount.

Strace of a setuptools install:
chmod("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg", 0644) = 0
listxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl\0", 256) = 16
getxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl", "\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00", 128) = 80
setxattr("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg", "system.nfs4_acl", "\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00", 80, 0) = -1 EIO (Input/output error)
stat("/tmp/easy_install-41z79x8r", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/tmp/easy_install-41z79x8r", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
getdents(4, /* 2 entries */, 32768)     = 48
getdents(4, /* 0 entries */, 32768)     = 0
close(4)                                = 0
rmdir("/tmp/easy_install-41z79x8r")     = 0
write(2, "error: [Errno 5] Input/output er"..., 125error: [Errno 5] Input/output error: '/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg'
) = 125
msg304970 - (view) Author: Albert Zeyer (Albert.Zeyer) Date: 2017-10-25 09:23
I'm also affected by this, with Python 3.6. My home directory is on a ZFS-backed NFS share.
See here for details:
https://github.com/Linuxbrew/homebrew-core/issues/4799

Basically:
Copying setuptools.egg-info to /u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info
error: [Errno 5] Input/output error: '/u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info/zip-safe'

Note that also by other tools, such as `mv` and `cp`, I get errors about setting `system.nfs4_acl`. But they just seem to ignore that and go on. I think this is the right thing to do here. You can print a warning about that, but then just go on. Maybe esp. just for `system.nfs4_acl`.
msg322848 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-08-01 07:43
[ https://bugs.python.org/issue24564#msg246278 ]

> Adding `EINVAL` to the ignored errnos would fix the problem, but might hide real failures (I'm not sure about the real failures, but it seems logical).

I think this is an acceptable compromise considering that:

1) because of this the copy operation will ultimately fail (despite file will be copied)
2) there is nothing we can do except emit a warning somehow 
3) EPERM and ENODATA are already silently ignored (also ENOTSUP but that is more legit)

For completeness, other than copytree() also copy2() and move() are affected. 
I submitted a PR: https://github.com/python/cpython/pull/8601.
msg322942 - (view) Author: desbma (desbma) * Date: 2018-08-02 10:51
> Since the `copy_function` is customizable to switch between `copy` and `copy2`, making copystat optional on files, perhaps the `copystat` should be optional on directories, as well.

Related: https://bugs.python.org/issue32073
msg341858 - (view) Author: hans.meine (hans-meine) Date: 2019-05-08 07:49
I completely agree with msg322848, both with the suggested fix and with its rationale.  A future API improvement could be to have a "strict" mode where errors copying xattr are not silently ignored anymore (if there are src xattr, but they cannot be written, or if src xattr cannot be read, for instance), but the current functionality seems to be "copy xattr if possible", and that should be fixed accordingly.

I also opened #36850 arguing that in any case, EINVAL should be ignored when *reading* extended attributes, acknowledging that people might have more problems with ignoring errors on *writing* them when they were present on the source.  However, as stated above, I personally vote for the fix suggested in msg322848.
msg342794 - (view) Author: Cheryl Sabella (cheryl.sabella) * (Python committer) Date: 2019-05-18 11:51
Nosying Hynek and Larry, as they had done the original code.
msg343944 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-30 03:25
New changeset a16387ab2d85f19665920bb6ff91a7e57f59dd2a by Giampaolo Rodola (Ying Wang) in branch 'master':
bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
https://github.com/python/cpython/commit/a16387ab2d85f19665920bb6ff91a7e57f59dd2a
msg343949 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2019-05-30 05:58
New changeset f1487b323549e2360460383b4304f6592fb38e27 by Giampaolo Rodola (Miss Islington (bot)) in branch '3.7':
bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
https://github.com/python/cpython/commit/f1487b323549e2360460383b4304f6592fb38e27
msg351555 - (view) Author: Michael Burt (Michael Burt) Date: 2019-09-09 21:58
This is still a problem when shutil gets a errno.ENOSYS

I hit this bug on Microsoft Azure when I mount an Azure File (managed NFS) into an AKS cluster (managed Kubernetes offering) and try to copy a file from the NFS over to the local disk on the node using shutil.copytree().

The workaround I am using came from this StackOverflow answer: https://stackoverflow.com/a/51635427/3736286
History
Date User Action Args
2019-09-09 21:58:13Michael Burtsetnosy: + Michael Burt
messages: + msg351555
2019-05-30 06:03:47giampaolo.rodolasetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: - Python 2.7
2019-05-30 05:58:46giampaolo.rodolasetmessages: + msg343949
2019-05-30 03:27:14miss-islingtonsetpull_requests: + pull_request13562
2019-05-30 03:25:39giampaolo.rodolasetmessages: + msg343944
2019-05-18 11:51:32cheryl.sabellasetnosy: + hynek, larry, cheryl.sabella

messages: + msg342794
versions: - Python 3.4, Python 3.5, Python 3.6
2019-05-16 23:38:55yingw787setkeywords: + patch
pull_requests: + pull_request13279
2019-05-08 07:49:39hans-meinesetnosy: + hans-meine
messages: + msg341858
2018-10-17 12:36:46cdyson37setnosy: + cdyson37
2018-08-02 10:51:22desbmasetnosy: + desbma
messages: + msg322942
2018-08-01 07:43:12giampaolo.rodolasetversions: + Python 2.7, Python 3.7, Python 3.8
nosy: + giampaolo.rodola

messages: + msg322848
pull_requests: + pull_request8109

stage: patch review
2018-07-11 07:50:28serhiy.storchakasettype: crash -> behavior
components: + Library (Lib)
2017-10-25 09:23:44Albert.Zeyersetnosy: + Albert.Zeyer

messages: + msg304970
versions: + Python 3.6
2016-11-04 15:16:22Gabriel Devenyisetnosy: + Gabriel Devenyi

messages: + msg280051
versions: + Python 3.5
2015-07-04 21:45:32r.david.murraysetnosy: + r.david.murray
messages: + msg246279
2015-07-04 21:19:23minrksetnosy: + minrk
messages: + msg246278
2015-07-04 20:26:59jhamricksetmessages: + msg246273
2015-07-04 20:05:50jhamrickcreate