I am using shutil.copytree to setup new user home directories within an automated system.  The copy2 function is called in order to copy individual files and preserve stat data. 

However, copytree simply calls os.mkdir and leaves directory creation at the mercy of my current umask (in my case, that's daemon context - 0).

I've got to then iterate through the newly copied tree and set permissions on each individual subdirectory. 

Adding a simple copystat(src, dst) on line 112 of shutil.py fixes the problem. 

The result should be uniform; either preserve permissions across the board, or leave it to the mercy of the caller.  I know there's an enhancement request already open to supply a 'func=' kw argument to copytree.

python -V
Python 2.4.3

on 

Linux marvin 2.6.18-1.2257.fc5smp #1 SMP Fri Dec 15 16:33:51 EST 2006 i686 i686 i386 GNU/Linux

I can confirm this bug.

For MoinMoin wiki, we solved that by duplicating this function to our own tree and we did exactly the proposed change:
 
Adding a simple copystat(src, dst) on line 112 of shutil.py (right below os.mkdir) fixes this problem.

I also suggest to remove that XXX comment in that function. With the proposed fix, it gets usable and thus is not only example code any more.

BUT: for avoiding problems on win32, a second fix needs to be done. For MoinMoin, we call this wrapper around shutil.copystat from our copytree function:

def copystat(src, dst):
    """Copy stat bits from src to dst

    This should be used when shutil.copystat would be used on directories
    on win32 because win32 does not support utime() for directories.

    According to the official docs written by Microsoft, it returns ENOACCES if the
    supplied filename is a directory. Looks like a trainee implemented the function.
    """
    if sys.platform == 'win32' and S_ISDIR(os.stat(dst)[ST_MODE]):
        if os.name == 'nt':
            st = os.stat(src)
            mode = S_IMODE(st[ST_MODE])
            if hasattr(os, 'chmod'):
                os.chmod(dst, mode) # KEEP THIS ONE!
        #else: pass # we are on Win9x,ME - no chmod here
    else:
        shutil.copystat(src, dst)

As you see, some special treatment is needed for win32/nt and directories - and directories are exactly the use case used by the fixed copytree, so a fixed copystat is needed or copytree will crash on win32/nt.

> The result should be uniform; either preserve permissions across the
> board, or leave it to the mercy of the caller.
I’m surprised by this report, as the code does a copystat(sourcedir, targetdir) since 2004 (#1048878).  Anyhow, I think the second option would be better.

>  I know there's an enhancement request already open to supply a
> 'func=' kw argument to copytree.
This is now implemented (copy_function argument), but does not help with directories, only with files.  Recently I needed to call copytree and change the permissions of the destination, so I wanted to use a two-line custom copy_function (one line to call shutil.copy, one line for os.chmod), but I hadn’t understood that the custom function was not used for directory creation.  I found out that shutil does a copystat on the created directories behind my back.  So, I think this problem can be solved in a nice and b/w-compatible way by adding a new keyword argument, makedirs_func.  I would like to fix this in stable versions too, but if another core developer argues that it should be considered a new feature, I won’t argue.

I made the change suggested in the last comment, patch is attached. Trying to clean up any bugs I've got my name on!

Patch looks good.

I don’t know if I should fix this in 3.3 with your patch or go back to the first idea of adding a copystat(src, dst) after the mkdir call.  I just don’t know if it’s important that this behavior does not change in stable versions, or if it’s clearly a bug and we can change it.

Ah, understood. I kind of like the idea of having the added functionality behind a custom callable, but if it's generally just a bug then copystat is a good solution, too.

Attaching a test that verifies that shutil.copytree is, in fact, preserving permissions (as of Python 3.4.0a0 (default:66a3dc613627, Jul 27 2013, 21:23:10) )

As far as I can tell this can be closed.

Catherine, I think it would be nice to apply your patch to avoid any further regressions. However, you should first sign a contributor's agreement (http://www.python.org/psf/contrib/). Could you please do so and post here once it is done?

Thanks, Antoine - I've signed and submitted the Contributor's Agreement.

Hi Catherine,

the contributor agreement must go through proper channels. The bug tracker is not the right place to post your agreement. Please follow the instructions at http://www.python.org/psf/contrib/ . You can submit the agreement electronically, too.

Hi Catherine,

I saw your patch. It looks good, except you are trying to copy the directory onto its subdirectory.

src_dir = tempfile.mkdtemp()
dst_dir = os.path.join(tempfile.mkdtemp(), 'destination')

I prefer it if you copy it to somewhere else. This would be better:

tmp_dir = self.mkdtemp()
src = os.path.join(tmp_dir, 'foo')
dst = os.path.join(tmp_dir, 'bar')

Hey Catherine,
Do you want to update your patch to include Vajrasky's suggestion?

Attaching new patch incorporating Vajrasky's suggestion

New changeset c388e93879c4 by Antoine Pitrou in branch '3.3':
Issue #1666318: Add a test that shutil.copytree() retains directory permissions.
http://hg.python.org/cpython/rev/c388e93879c4

New changeset 8906713d5704 by Antoine Pitrou in branch 'default':
Issue #1666318: Add a test that shutil.copytree() retains directory permissions.
http://hg.python.org/cpython/rev/8906713d5704

Patch committed to 3.3 and 3.4. Thank you very much!