classification
Title: distutils: ignore .nfsXXXX files
Type: behavior Stage: resolved
Components: Distutils, Distutils2 Versions: Python 3.4, Python 3.2, Python 3.3, Python 2.7, 3rd party
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: eric.araujo Nosy List: SilentGhost, eric.araujo, grobian, haubi, jramnani, neologix, python-dev, sandro.tosi, tarek, zbysz
Priority: normal Keywords: easy, needs review, patch

Created on 2010-01-16 19:16 by grobian, last changed 2012-12-09 04:02 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
python-2.5.1-distutils-aixnfs.patch grobian, 2010-01-16 19:16 distutils: ignore .nfsXXXXX files
dir_util.py.diff SilentGhost, 2011-02-07 14:43 review
issue7719-nfs_silly_rename.patch jramnani, 2012-03-13 21:37 review
issue7719-nfs_silly_rename.patch jramnani, 2012-05-22 04:07 distutils: ignore .nfsXXXXX files review
Messages (18)
msg97898 - (view) Author: Fabian Groffen (grobian) Date: 2010-01-16 19:16
NFS on certain platforms (most notably AIX, Solaris) use .nfsXXXXX files that appear and disappear automagically.  distutils can get confused by that once a .nfsXXXXX file was earlier seen with listdir and then removed by the OS before its copied.

Simply ignore .nfsXXXXX files as workaround.
msg128118 - (view) Author: SilentGhost (SilentGhost) * Date: 2011-02-07 14:43
Here is the applicable patch against py3k branch.
msg128193 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-02-08 22:40
Thanks for the report and patch (I think SilentGhost’s version is better).  Do you have any pointer about those .nfs* files?  Are there other (build) tools that ignore them?  Is it always safe to skip .nfs* files, or only .nfs????? (i.e. six characters)?

Aside from those questions, a patch requires a test and doc update.  See guidelines at http://wiki.python.org/moin/Distutils/FixingBugs
msg128195 - (view) Author: Sandro Tosi (sandro.tosi) * (Python committer) Date: 2011-02-08 22:45
On Tue, Feb 8, 2011 at 23:40, Éric Araujo <report@bugs.python.org> wrote:
>
> Éric Araujo <merwok@netwok.org> added the comment:
>
> Do you have any pointer about those .nfs* files?  Are there other (build) tools that ignore them?  Is it always safe to skip .nfs* files, or only .nfs????? (i.e. six characters)?

Just replying for this part: .nfs* files are created by the nfs server
when on of its client removes a file while another client has the very
same file opened. the nfs server keeps the .nfs* file around until the
last client closes the file, after that it removes the .nfs* file.

About the number of digits after '.nfs' I'm not sure there's a
standard for it, but I saw files with more that 6 chars after that.

Regards,
-- 
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi
msg150348 - (view) Author: Zbyszek Jędrzejewski-Szmek (zbysz) * Date: 2011-12-29 20:39
Review of both patches (python-2.5.1-distutils-aixnfs.patch and dir_util.py.diff) as they are essentially the same:

I think that the test is in wrong place: we would want to ignore those .nfs* files always, not just when checking for symlinks. A separate test
at the top of the loop would be better:
    for n in names:
+       if n.startswith('.nfs'):
+           continue
        src_name = os.path.join(src, n)
        dst_name = os.path.join(dst, n)
        if preserve_symlinks and os.path.islink(src_name):


BTW: under linux 2.6.20 I see files like: .nfs000000000592413900000036, always of this length, with hexadecimal digits after .nfs).
msg153689 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-02-19 09:51
Thanks to Sandro’s explanation and a passage in the Unix Haters Handbook, I now think that ignoring .nfs* files is a safe change.  If should not be hard to add a test for this in test_sdist.
msg155673 - (view) Author: Jeff Ramnani (jramnani) * Date: 2012-03-13 21:37
I've updated SilentGhost's patch to include a test and documentation.
msg159688 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-04-30 11:06
Jeff,

the patch LGTM but Eric indicated he'd like to have a higher level test inside Lib/distutils/tests/test_sdist.py. Possibly as part of a bigger test like test_prune_file_list or test_add_defaults; no need to remove the old test though I guess.

P.S. There's an extra empty line inside of apiref.rst. You may want to strip it from your next patch.
msg160854 - (view) Author: Hynek Schlawack (hynek) * (Python committer) Date: 2012-05-16 12:45
Jeff, are you still interested in updating your patch or would you prefer if someone else tackled it?
msg160860 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2012-05-16 13:41
Wouldn't it be better to add an 'ignore' option to the copy_tree() method with an optional list of patterns to ignore instead of hardcoding '.nsfXXX' files?
This would make it possible to also skip '.hg', 'CVS' artifacts.
msg160893 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-16 17:47
No improvements are done in distutils, only the smallest possible changes to fix bugs.  In distutils2 we use shutil.copytree.
msg161318 - (view) Author: Jeff Ramnani (jramnani) * Date: 2012-05-22 02:48
Hynek,

I would indeed like to update this patch.  I will make the updates you have suggested.
msg161319 - (view) Author: Jeff Ramnani (jramnani) * Date: 2012-05-22 04:07
Attached the updated patch.  I have added a .nfs file to test_prune_file_list in test_sdist.  I've also cleaned up the documentation to read a little better (IMO).
msg161321 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-05-22 05:02
Thanks, will apply.
msg164570 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-03 05:02
SilentGhost, could you submit a contributor agreement to the PSF for your patches?  See http://www.python.org/psf/contrib/ .  Thanks.
msg164571 - (view) Author: Roundup Robot (python-dev) Date: 2012-07-03 05:14
New changeset a56cebff113a by Éric Araujo in branch '2.7':
Ignore .nfs* files in distutils (#7719).
http://hg.python.org/cpython/rev/a56cebff113a
msg177192 - (view) Author: Roundup Robot (python-dev) Date: 2012-12-09 03:57
New changeset d978828bfd0e by Éric Araujo in branch '3.2':
Ignore .nfs* files in distutils (#7719).
http://hg.python.org/cpython/rev/d978828bfd0e

New changeset 10ab746f55fb by Éric Araujo in branch '3.3':
Merge fixes for #13614, #13512 and #7719 from 3.2
http://hg.python.org/cpython/rev/10ab746f55fb

New changeset b10c1c6f869f by Éric Araujo in branch 'default':
Merge fixes for #13614, #13512 and #7719 from 3.3
http://hg.python.org/cpython/rev/b10c1c6f869f
msg177195 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-12-09 04:02
Thanks all!
History
Date User Action Args
2012-12-09 04:02:20eric.araujosetstatus: open -> closed
versions: + Python 3.4
messages: + msg177195

resolution: fixed
stage: commit review -> resolved
2012-12-09 03:57:19python-devsetmessages: + msg177192
2012-07-03 05:14:02python-devsetnosy: + python-dev
messages: + msg164571
2012-07-03 05:02:42eric.araujosetmessages: + msg164570
2012-06-24 07:32:55hyneksetnosy: - hynek
2012-05-22 05:02:13eric.araujosetmessages: + msg161321
stage: test needed -> commit review
2012-05-22 04:07:34jramnanisetfiles: + issue7719-nfs_silly_rename.patch

messages: + msg161319
2012-05-22 02:48:50jramnanisetmessages: + msg161318
2012-05-16 17:47:30eric.araujosetmessages: + msg160893
2012-05-16 13:41:10neologixsetnosy: + neologix
messages: + msg160860
2012-05-16 12:45:46hyneksetmessages: + msg160854
2012-04-30 11:06:14hyneksetnosy: + hynek
messages: + msg159688
2012-03-13 21:37:04jramnanisetfiles: + issue7719-nfs_silly_rename.patch
nosy: + jramnani
messages: + msg155673

2012-02-19 09:51:44eric.araujosetkeywords: + easy
assignee: tarek -> eric.araujo
messages: + msg153689

versions: + 3rd party
2011-12-29 20:39:47zbyszsetnosy: + zbysz

messages: + msg150348
versions: + Python 3.3, - 3rd party, Python 3.1
2011-02-08 22:45:51sandro.tosisetnosy: + sandro.tosi
messages: + msg128195
2011-02-08 22:40:56eric.araujosetversions: + 3rd party
nosy: + eric.araujo

messages: + msg128193

components: + Distutils2
2011-02-07 14:43:36SilentGhostsetfiles: + dir_util.py.diff
nosy: + SilentGhost
messages: + msg128118

2011-02-07 13:40:21haubisetnosy: + haubi
2010-07-11 12:10:35BreamoreBoysetversions: + Python 3.1, Python 3.2, - Python 2.6, Python 2.5
2010-01-16 19:36:24brian.curtinsetpriority: normal
keywords: + needs review
stage: test needed
2010-01-16 19:16:43grobiancreate