classification
Title: Add copy_directory_metadata parameter to shutil.copytree
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: desbma, giampaolo.rodola
Priority: normal Keywords: patch

Created on 2017-11-18 19:13 by desbma, last changed 2018-08-03 21:34 by desbma.

Pull Requests
URL Status Linked Edit
PR 4456 open desbma, 2017-11-18 19:27
Messages (7)
msg306494 - (view) Author: desbma (desbma) * Date: 2017-11-18 19:13
I am sometimes using shutil.copytree to copy a directory to a destination that does not support setting metadata (like MTP mounts of Android devices).

Using the copy_function parameter allows passing shutil.copy or a custom function to ignore file metadata, however currently shutil.copytree always tries to call copystat on directories, which can fail with OSError (errno set to ENOTSUPP).

The code assumes copystat can fail on Windows, but propagates the error on other OS, even though the tree copy actually succeeds.
See https://github.com/python/cpython/blob/9bb6fe52742340f6c92f0dda18599a4577a94e18/Lib/shutil.py#L352-L357
msg308628 - (view) Author: desbma (desbma) * Date: 2017-12-19 11:49
Ping
msg323055 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-08-03 14:36
What function raises ENOTSUPP exactly (traceback would be welcome)?
msg323075 - (view) Author: desbma (desbma) * Date: 2018-08-03 20:56
Traceback is not very useful in that case:

mkdir /tmp/a
touch /tmp/a/b 

python
Python 3.6.6 (default, Jun 27 2018, 13:11:40) 
[GCC 8.1.1 20180531] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, shutil
>>> os.getcwd()
'/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1'
>>> shutil.copytree("/tmp/a", os.path.join(os.getcwd(), "test"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/desbma/py-venvs/main/lib/python3.6/shutil.py", line 359, in copytree
    raise Error(errors)
shutil.Error: [('/tmp/a/b', '/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1/test/b', '[Errno 95] Operation not supported'), ('/tmp/a', '/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1/test', '[Errno 95] Operation not supported')]
>>> shutil.copytree("/tmp/a", os.path.join(os.getcwd(), "test2"), copy_function=shutil.copy)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/desbma/py-venvs/main/lib/python3.6/shutil.py", line 359, in copytree
    raise Error(errors)
shutil.Error: [('/tmp/a/b', '/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1/test2/b', "[Errno 95] Operation not supported: '/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1/test2/b'"), ('/tmp/a', '/run/user/1000/gvfs/mtp:host=%5Busb%3A001%2C006%5D/sdcard1/test2', '[Errno 95] Operation not supported')]

The exception is thrown from here https://github.com/python/cpython/blob/9bb6fe52742340f6c92f0dda18599a4577a94e18/Lib/shutil.py#L359
msg323077 - (view) Author: desbma (desbma) * Date: 2018-08-03 21:01
Note that in the examples above both copytree calls actually succeed (only metadata copy failed).

The user can disable file metadata copy by passing 'copy_function=shutil.copy', but there is no way to do the same for directories and the directory copystat call https://github.com/python/cpython/blob/9bb6fe52742340f6c92f0dda18599a4577a94e18/Lib/shutil.py#L353 fails.
msg323081 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2018-08-03 21:25
It's not clear where the exception originates from. Try to use copy2() instead of copytree().
msg323082 - (view) Author: desbma (desbma) * Date: 2018-08-03 21:34
copy2 always raises "OSError: [Errno 95] Operation not supported" here https://github.com/python/cpython/blob/9bb6fe52742340f6c92f0dda18599a4577a94e18/Lib/shutil.py#L258 but I can work around that by passing copy_function=shutil.copy to copytree as I did above.

The problem is that there is currently no way to avoid the copystat call made on the directory itself.
History
Date User Action Args
2018-08-03 21:34:55desbmasetmessages: + msg323082
2018-08-03 21:25:43giampaolo.rodolasetmessages: + msg323081
2018-08-03 21:01:03desbmasetmessages: + msg323077
2018-08-03 20:56:38desbmasetmessages: + msg323075
2018-08-03 14:36:57giampaolo.rodolasetmessages: + msg323055
2018-06-12 09:41:30giampaolo.rodolasetnosy: + giampaolo.rodola
2017-12-19 11:49:25desbmasetmessages: + msg308628
2017-11-18 19:27:13desbmasetkeywords: + patch
stage: patch review
pull_requests: + pull_request4394
2017-11-18 19:13:27desbmacreate