classification
Title: shutil.move doesn't handle ENOTSUP raised by chflags on OS X
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: hynek Nosy List: eric.araujo, grobian, hynek, ned.deily, pitrou, python-dev
Priority: normal Keywords: patch

Created on 2012-04-24 17:58 by grobian, last changed 2012-05-11 01:20 by ned.deily. This issue is now closed.

Files
File name Uploaded Description Edit
expand-chflags-catch-2.7.diff hynek, 2012-04-27 17:30 Patch to catch also ENOTSUP in copystat() for Python 2.7 review
expand-chflags-catch-tip.diff hynek, 2012-04-27 17:55 review
expand-chflags-catch-3.2.diff hynek, 2012-04-27 17:56 review
expand-chflags-catch-2.7-v2.diff hynek, 2012-05-02 15:27 review
expand-chflags-catch-3.2-v2.diff hynek, 2012-05-02 15:36 review
expand-chflags-catch-tip-v2.diff hynek, 2012-05-02 15:47 review
Messages (22)
msg159178 - (view) Author: Fabian Groffen (grobian) Date: 2012-04-24 17:58
With current working dir an NFS-mounted ZFS share, and /var/tmp (OSX default) HFS+:

% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 24 2012, 19:33:45) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% ls /var/tmp/testfile ./testfile
./testfile  /var/tmp/testfile

The problem likely is that the flags stored on the HFS+ volume cannot be applied to the NFS-mounted ZFS volume.  This likely also occurs when doing a bit more regular things, like e.g. moving/copying to a mounted USB disk (with FAT32 filesystem).

I believe this is a "regression" introduced by http://bugs.python.org/issue8746.  Python-2.7.2 works fine.

While preserving flags is nice, it is questionable whether failure to do so in this case is worth dying for.  In particular, leaving behind both the original as well as the copy is a bit messy.
msg159184 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-24 18:22
I guess a “best effort” approach would be best here.

I presume Python 3.2+ have the same behavior?
msg159185 - (view) Author: Fabian Groffen (grobian) Date: 2012-04-24 18:26
> I presume Python 3.2+ have the same behavior?

I cannot compile that or get it working normally, so I can't tell for sure.  Judging from the code, I'd say yes.
msg159192 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-24 19:08
Now that’s odd. I just looked into the code at http://hg.python.org/cpython/file/2.7/Lib/shutil.py#l103 and there is a guard against EOPNOTSUPP:

try:
   os.chflags(dst, st.st_flags)
except OSError, why:
   if (not hasattr(errno, 'EOPNOTSUPP') or
   why.errno != errno.EOPNOTSUPP):
      raise

Does your /Library/Gentoo/usr/lib/python2.7/shutil.py look the same? Would you mind adding a `print why.errno` just before the raise?

I have tried move'ing files to NTFS and FAT32 and it works just fine here.
msg159405 - (view) Author: Fabian Groffen (grobian) Date: 2012-04-26 17:42
% echo "test" > /var/tmp/testfile
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 299, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 129, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% vi /Library/Gentoo/usr/lib/python2.7/shutil.py
% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import shutil
>>> shutil.move("/var/tmp/testfile", "./testfile");
45
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 300, in move
    copy2(src, real_dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 130, in copy2
    copystat(src, dst)
  File "/Library/Gentoo/usr/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 45] Operation not supported: './testfile'
>>> 
% grep 45 /usr/include/sys/errno.h
#define ENOTSUP         45              /* Operation not supported */

I tried with a FAT16 formatted USB-disk, but there it doesn't fail, so I did some further digging.  MS-DOS FS (under OSX) just seems to support setting flags (I tried with stat.UF_HIDDEN, Finder no longer displays the file).

NFS, however, does NOT support any chflags call.

% python
Python 2.7.3 (default, Apr 26 2012, 19:06:37) 
[GCC 4.2.1 (Gentoo 4.2.1_p5666, Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import errno
>>> print hasattr(errno, 'EOPNOTSUPP')
True
>>> print errno.EOPNOTSUPP
102
>>> 

102 obviously != 45

% grep 102 /usr/include/sys/errno.h
#define EOPNOTSUPP      102             /* Operation not supported on socket */

I believe Python got it mixed up here, we're looking for ENOTSUP, but that one doesn't exist, at least not here.

>>> print hasattr(errno, 'ENOTSUP')
False
msg159407 - (view) Author: Fabian Groffen (grobian) Date: 2012-04-26 17:51
it seems errnomodule.c has no idea of ENOTSUP, and that's not the only missing one.

OSX 10.7:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ENOTSUP
missing: EBADRPC
missing: ERPCMISMATCH
missing: EPROGUNAVAIL
missing: EPROGMISMATCH
missing: EPROCUNAVAIL
missing: EFTYPE
missing: EAUTH
missing: ENEEDAUTH
missing: EPWROFF
missing: EDEVERR
missing: EBADEXEC
missing: EBADARCH
missing: ESHLIBVERS
missing: EBADMACHO
missing: ECANCELED
missing: ENOATTR
missing: ENOPOLICY
missing: ENOTRECOVERABLE
missing: EOWNERDEAD
missing: ELAST

Solaris 10:
$ grep "^#define\sE" /usr/include/sys/errno.h | awk '{print $2}' | while read line ; do grep -q ${line} Modules/errnomodule.c || echo "missing: $line" ; done
missing: ECANCELED
missing: ENOTSUP
missing: EOWNERDEAD
missing: ENOTRECOVERABLE
missing: ELOCKUNMAPPED
missing: ENOTACTIVE
msg159417 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-04-26 19:34
Thanks for the analysis.  Yes, it looks like there's a difference between OS X and current FreeBSDs, for example.  chflags(2) on the latter is documented as returning EOPNOTSUPP and on the former ENOTSUP. shutil should check for both.  A quick search of the source tree did not find any other users of chflags in the standard library. As far as adding other missing errnos, that could be handled as a separate issue as it more of an enhancement.  Anyone interested in contributing a patch for either or both?

https://developer.apple.com/library/mac/#documentation/darwin/reference/manpages/man2/chflags.2.html

http://www.freebsd.org/cgi/man.cgi?query=chflags&apropos=0&sektion=2&manpath=FreeBSD+9.0-RELEASE&arch=default&format=html
msg159419 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-26 19:46
I had this text ready before ned chimed in, I’ll post it anyway because it was a lot of work ;):

You're right, 2.7’s errnos are incomplete compared to 3.2. Antoine added ENOTSUP in c370866f30a7 and it runs as “Solaris-specific”.

So it’s currently in 3.2 and later. Shouldn’t hurt to back port it?

EOPNOTSUP is obviously wrong in your case and it doesn’t really sound right at all by the description. However, maybe on some other architecture (FreeBSD?) it’s the way to go? The commit (2e0d58adadbe) states it’s because of ZFS on OS X.

As the code is unchanged in 3.2+, this bug also applies to them.


Suggestion:

For 3.2+3.3: I’d extend the catch to also catch ENOTSUP
For 2.7: I’d also backport the err code.

NB I’m fine if Fabian wants to do it himself, it’s his issue.
msg159422 - (view) Author: Fabian Groffen (grobian) Date: 2012-04-26 20:25
I don't want to go through the paperwork nonsense just for a trivial patch, hence I didn't supply one, but instead provided all the information for you guys to make the correct fix.
msg159423 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-04-26 20:32
Trivial patches don’t require paperwork; non-trivial patches require a simple contributor agreement (print, sign, scan, email).  We don’t like that either but it is required.  If you have any suggestion to make the process simpler, please share them on python-dev.
msg159479 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-27 17:30
This is a fix for 2.7.

As Benjamin said in http://bugs.python.org/issue14682#msg159477 it’s okay to back port ENOTSUP, I did it as part of the patch here. I wasn’t sure whether we should document it?

I’m porting the patch to tip right now, reviews/opinions welcome.
msg159480 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-27 17:55
This one is against tip.
msg159481 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-27 17:56
And finally against 3.2
msg159737 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-04-30 23:07
> I wasn’t sure whether we should document it?

No, it should remain "hidden".
msg159795 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-02 13:30
The test fails here (Linux), since there's no os.chflags:

======================================================================
ERROR: test_copystat_handles_harmless_chflags_errors (test.test_shutil.TestShutil)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/antoine/cpython/default/Lib/test/test_shutil.py", line 302, in test_copystat_handles_harmless_chflags_errors
    old_chflags = os.chflags
AttributeError: 'module' object has no attribute 'chflags'
msg159796 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-05-02 15:27
Fixed for 2.7
msg159798 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-05-02 15:36
Fixed for 3.2.
msg159803 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-05-02 15:47
And finally tip.
msg159947 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-05-04 17:29
Looks ok to me, but I don't have a system with os.chflags to test on.
msg159970 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-05-04 20:17
I will test and check it in next week if still open.
msg160395 - (view) Author: Roundup Robot (python-dev) Date: 2012-05-11 01:14
New changeset e12efebc3ba6 by Ned Deily in branch '2.7':
Issue #14662: Prevent shutil failures on OS X when destination does not
http://hg.python.org/cpython/rev/e12efebc3ba6

New changeset ae141eebcf96 by Ned Deily in branch '3.2':
Issue #14662: Prevent shutil failures on OS X when destination does not
http://hg.python.org/cpython/rev/ae141eebcf96

New changeset 93599d5e0a23 by Ned Deily in branch 'default':
Issue #14662: Prevent shutil failures on OS X when destination does not
http://hg.python.org/cpython/rev/93599d5e0a23
msg160396 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2012-05-11 01:20
Thanks for the patch!  Tested with an NFS-mounted file system on OS X.  Applied for 2.7.4, 3.2.4, and 3.3.0.
History
Date User Action Args
2012-05-11 01:20:47ned.deilysetstatus: open -> closed
resolution: fixed
messages: + msg160396

stage: patch review -> resolved
2012-05-11 01:14:33python-devsetnosy: + python-dev
messages: + msg160395
2012-05-04 20:17:39ned.deilysetmessages: + msg159970
2012-05-04 17:29:13pitrousetmessages: + msg159947
2012-05-02 15:47:01hyneksetfiles: + expand-chflags-catch-tip-v2.diff

messages: + msg159803
2012-05-02 15:36:54hyneksetfiles: + expand-chflags-catch-3.2-v2.diff

messages: + msg159798
2012-05-02 15:27:51hyneksetfiles: + expand-chflags-catch-2.7-v2.diff

messages: + msg159796
2012-05-02 13:30:33pitrousetmessages: + msg159795
2012-05-01 11:41:06hyneksetassignee: hynek
title: shutil.move broken in 2.7.3 on OSX (chflags fails) -> shutil.move doesn't handle ENOTSUP raised by chflags on OS X
2012-04-30 23:07:45pitrousetmessages: + msg159737
2012-04-29 15:49:10hyneksetcomponents: + Library (Lib), - None
stage: needs patch -> patch review
2012-04-27 17:56:19hyneksetfiles: + expand-chflags-catch-3.2.diff

messages: + msg159481
2012-04-27 17:55:52hyneksetfiles: + expand-chflags-catch-tip.diff

messages: + msg159480
2012-04-27 17:30:05hyneksetfiles: + expand-chflags-catch-2.7.diff

nosy: + pitrou
messages: + msg159479

keywords: + patch
2012-04-26 20:32:16eric.araujosetnosy: + eric.araujo
messages: + msg159423
2012-04-26 20:25:13grobiansetmessages: + msg159422
2012-04-26 19:46:43hyneksetmessages: + msg159419
2012-04-26 19:34:54ned.deilysetversions: + Python 3.2, Python 3.3
nosy: + ned.deily

messages: + msg159417

stage: needs patch
2012-04-26 17:51:16grobiansetmessages: + msg159407
2012-04-26 17:42:19grobiansetmessages: + msg159405
2012-04-24 19:08:15hyneksetmessages: + msg159192
2012-04-24 18:26:45grobiansetmessages: + msg159185
2012-04-24 18:22:35hyneksetnosy: + hynek
messages: + msg159184
2012-04-24 17:58:51grobiancreate