classification
Title: shutil.copytree() crashes copying to VFAT on Linux: AttributeError: 'PermissionError' object has no attribute 'winerror'
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.5, Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gward Nosy List: berker.peksag, gward, haypo, python-dev, serhiy.storchaka
Priority: normal Keywords: 3.4regression

Created on 2014-06-15 21:37 by gward, last changed 2014-12-10 13:11 by haypo. This issue is now closed.

Messages (10)
msg220676 - (view) Author: Greg Ward (gward) (Python committer) Date: 2014-06-15 21:37
When using shutil.copytree() on Linux to copy to a VFAT filesystem, it crashes like this:

Traceback (most recent call last):
  File "/data/src/cpython/3.4/Lib/shutil.py", line 336, in copytree
    copystat(src, dst)
  File "/data/src/cpython/3.4/Lib/shutil.py", line 190, in copystat
    lookup("chmod")(dst, mode, follow_symlinks=follow)
PermissionError: [Errno 1] Operation not permitted: '/mnt/example_nt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "copytree-crash.py", line 14, in <module>
    shutil.copytree('PC/example_nt', '/mnt/example_nt')
  File "/data/src/cpython/3.4/Lib/shutil.py", line 339, in copytree
    if why.winerror is None:
AttributeError: 'PermissionError' object has no attribute 'winerror'

I am *not* complaining about the PermissionError. That has been issue1545. Rather, I'm complaining about the crash that happens while attempting to handle the PermissionError.

Reproducing this is fairly easy, although it requires root privilege.

1. dd if=/dev/zero of=dummy bs=1024 count=1024
2. mkfs.vfat -v dummy
3. sudo mount -o loop /tmp/dummy /mnt

Then create the reproduction script in the root of Python's source dir:

cat > copytree-crash.py <<EOF
import os
import shutil

# assumptions:
#   1. /mnt is a directory writeable by current user
#   2. /mnt is a VFAT filesystem
#   3. current OS is Linux-ish

if os.path.exists('/mnt/example_nt'):
    print('cleaning up')
    shutil.rmtree('/mnt/example_nt')

print('copying')
shutil.copytree('PC/example_nt', '/mnt/example_nt')
EOF

and run it:

sudo ./python copytree-crash.py

Crash happens as documented above.
msg220677 - (view) Author: Greg Ward (gward) (Python committer) Date: 2014-06-15 21:40
In 3.3 and earlier, copytree() crashes roughly as described in issue1545, with shutil.Error that wraps the underlying "Operation not permitted" error from trying to chmod() something in a VFAT filesystem. Since this appears to accurately reflect what's coming from the kernel, I don't *think* this is a bug. Only the AttributeError, which is new in 3.4, is clearly a bug.
msg220679 - (view) Author: Greg Ward (gward) (Python committer) Date: 2014-06-15 21:44
Bad news: because reproducing this requires sudo (to mount an arbitrary filesystem), I'm not sure it's possible/desirable to add test code for it.

Good news: the fix is trivial, and it passes my manual test. Here's a patch:

--- a/Lib/shutil.py
+++ b/Lib/shutil.py
@@ -336,7 +336,7 @@
         copystat(src, dst)
     except OSError as why:
         # Copying file access times may fail on Windows
-        if why.winerror is None:
+        if getattr(why, 'winerror', None) is None:
             errors.append((src, dst, str(why)))
     if errors:
         raise Error(errors)

Running test suite now to make sure this doesn't break anything else...
msg231340 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-11-18 20:19
LGTM. Go ahead Greg.
msg231342 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-11-18 20:42
Would it be possible to write a unit test, maybe using unittest.mock to mock most parts?
msg231463 - (view) Author: Greg Ward (gward) (Python committer) Date: 2014-11-21 01:37
> Would it be possible to write a unit test, maybe using unittest.mock to 
> mock most parts?

Good idea! Turns out this was quite straightforward. The test patch is:

--- a/Lib/test/test_shutil.py
+++ b/Lib/test/test_shutil.py
@@ -1,6 +1,7 @@
 # Copyright (C) 2003 Python Software Foundation
 
 import unittest
+import unittest.mock
 import shutil
 import tempfile
 import sys
@@ -758,6 +759,20 @@
         self.assertEqual(os.stat(restrictive_subdir).st_mode,
                           os.stat(restrictive_subdir_dst).st_mode)
 
+    @unittest.mock.patch('os.chmod')
+    def test_copytree_winerror(self, mock_patch):
+        # When copying to VFAT, copystat() raises OSError. On Windows, the
+        # exception object has a meaningful 'winerror' attribute, but not
+        # on other operating systems. Do not assume 'winerror' is set.
+        src_dir = tempfile.mkdtemp()
+        dst_dir = os.path.join(tempfile.mkdtemp(), 'destination')
+        self.addCleanup(shutil.rmtree, src_dir)
+        self.addCleanup(shutil.rmtree, os.path.dirname(dst_dir))
+
+        mock_patch.side_effect = PermissionError('ka-boom')
+        with self.assertRaises(shutil.Error):
+            shutil.copytree(src_dir, dst_dir)
+
     @unittest.skipIf(os.name == 'nt', 'temporarily disabled on Windows')
     @unittest.skipUnless(hasattr(os, 'link'), 'requires os.link')
     def test_dont_copy_file_onto_link_to_itself(self):


When run without the bug fix, this reproduces my original failure nicely:

$ ./python Lib/test/test_shutil.py
................s........................s.s........E..s..............................s..
======================================================================
ERROR: test_copytree_winerror (__main__.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/src/cpython/3.4/Lib/shutil.py", line 337, in copytree
    copystat(src, dst)
  File "/data/src/cpython/3.4/Lib/shutil.py", line 191, in copystat
    lookup("chmod")(dst, mode, follow_symlinks=follow)
  File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 896, in __call__
    return _mock_self._mock_call(*args, **kwargs)
  File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 952, in _mock_call
    raise effect
PermissionError: ka-boom

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/src/cpython/3.4/Lib/unittest/mock.py", line 1136, in patched
    return func(*args, **keywargs)
  File "Lib/test/test_shutil.py", line 774, in test_copytree_winerror
    shutil.copytree(src_dir, dst_dir)
  File "/data/src/cpython/3.4/Lib/shutil.py", line 340, in copytree
    if why.winerror is None:
AttributeError: 'PermissionError' object has no attribute 'winerror'

----------------------------------------------------------------------
Ran 89 tests in 0.095s

FAILED (errors=1, skipped=5)

Excellent! No need for root privs, and the bug is quite obvious.

Apply my getattr() fix, and the test passes.

I'll commit on branch 3.4 and merge to default.
msg231464 - (view) Author: Greg Ward (gward) (Python committer) Date: 2014-11-21 01:55
> I'll commit on branch 3.4 and merge to default.

Whoops, never mind. Looks like I don't have push permission to hg.python.org after all. It's been 8 years since my last commit, so I shouldn't complain.

So... can someone with commit privs please 

  hg pull -r 3a1fc6452340 http://hg.gerg.ca/cpython

then merge to trunk and push?

Thanks!
msg232408 - (view) Author: Roundup Robot (python-dev) Date: 2014-12-10 00:51
New changeset 50517a4d7cce by Berker Peksag in branch '3.4':
Issue #21775: shutil.copytree(): fix crash when copying to VFAT
https://hg.python.org/cpython/rev/50517a4d7cce

New changeset 7d5754af95a9 by Berker Peksag in branch 'default':
Issue #21775: shutil.copytree(): fix crash when copying to VFAT
https://hg.python.org/cpython/rev/7d5754af95a9
msg232409 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2014-12-10 00:55
Committed the patch with a NEWS entry. Thanks Greg.

(You can send your SSH key to hgaccounts@python.org. See also https://docs.python.org/devguide/coredev.html#ssh)
msg232425 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-12-10 13:11
Note: Python 2.7 is not affected, I cannot find "winerror" in shutil.py.
History
Date User Action Args
2014-12-10 13:11:01hayposetmessages: + msg232425
2014-12-10 00:55:36berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg232409

resolution: fixed
stage: commit review -> resolved
2014-12-10 00:51:29python-devsetnosy: + python-dev
messages: + msg232408
2014-11-21 01:55:48gwardsetmessages: + msg231464
2014-11-21 01:37:30gwardsetmessages: + msg231463
2014-11-18 20:42:12hayposetnosy: + haypo
messages: + msg231342
2014-11-18 20:20:06serhiy.storchakasettype: behavior
2014-11-18 20:19:50serhiy.storchakasetversions: + Python 3.5
nosy: + serhiy.storchaka

messages: + msg231340

assignee: gward
stage: commit review
2014-06-15 21:44:19gwardsetmessages: + msg220679
2014-06-15 21:40:21gwardsetkeywords: + 3.4regression

messages: + msg220677
2014-06-15 21:37:40gwardcreate