classification
Title: distutils should not assume that hardlinks will work
Type: behavior Stage: resolved
Components: Distutils Versions: Python 3.5, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Matt.Wright, benjamin.peterson, dfarrell07, eric.araujo, fkochem, georg.brandl, larry, lemburg, olliewalsh, pitrou, python-dev, samtygier, sorin, tarek
Priority: normal Keywords: patch

Created on 2010-06-02 13:38 by samtygier, last changed 2014-10-30 19:14 by eric.araujo. This issue is now closed.

Files
File name Uploaded Description Edit
hardlink-traceback.txt samtygier, 2010-06-02 13:38
distutil-hardlink-trunk.diff samtygier, 2010-06-02 14:36 review
copy_file_link_fail.patch pitrou, 2014-10-29 20:25 review
Messages (24)
msg106881 - (view) Author: samtygier (samtygier) Date: 2010-06-02 13:38
distutils will currently try to use hardlinks if os has a 'link' attribute, however sometimes os.link() will fail, for example the filesystem may not support it (see attached traceback).

in commands/sdist.py in make_release_tree() there is the test:
        if hasattr(os, 'link'):        # can make hard links on this system
            link = 'hard'
            msg = "making hard links in %s..." % base_dir

'link' is then passed to copy_file() in file_util.py, which trusts that if link == 'hard', then hardlinking will work.

there has been discussion in the past, but i dont think it has been fixed
http://thread.gmane.org/gmane.comp.python.distutils.devel/2076
msg106885 - (view) Author: samtygier (samtygier) Date: 2010-06-02 14:36
here is a patch against http://svn.python.org/projects/python/trunk

it moves the return statements into the individual file copying sections, and takes the final call to _copy_file_contents() out of the else. this allows an error in hardlinking to fall through by passing the exception.

i have only tested on Linux.

when running it still prints out
"hard linking CHANGES -> pyzgoubi-0.4dev"
because of how the message is generated in line 130. do you think this needs to be changed. do we need a
"hardlinking failed, falling back to copy"
message? it might cause unnecessary worry (distutils already suppresses tracebacks to avoid scaring non-python-programmers).

i have a patch that applies to 2.6 as well, is there much chance of that being accepted?
msg106886 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2010-06-02 14:43
Distutils is a special case: Many third-party code relies on its undocumented quirks and bugs, so it’s basically frozen. Some non-disruptive bugfixes are accepted, in which case the normal Python rules apply (e.g., no new features in 2.7 which is in beta). Work has been moved to Distutils2, which breaks compatibility in order to fix things and add needed features.

If Tarek (maintainer of both packages) agrees this can go into Distutils, Benjamin Peterson (release manager) will say whether he agrees or not. Patient a while for Tarek to see this bug report, and thanks for your work :)
msg164543 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-02 20:37
AFAICT using hard links only serves to save up a little time and disk space; it seems to me that always copying would solve one or two bugs at a small cost (not so small for large projects, I don’t know).  Could this impact setup scripts?  Maybe if os.link were monkey-patched and expected to be called.  I could ask on distutils-sig.

In #15205 Ollie noted this: “distutils2 appears to always copy instead of hardlinking resolving all of these issues”.  That would be because d2 uses shutil.copytree and distutils.util.copy_file (yes, a function from distutils1 because of some Windows issue :().
msg164545 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-02 20:44
Of course the conservative fix would be to try linking as of now and do a copy when an exception is caught, as was proposed.

FTR python.org link to the thread mentioned: http://mail.python.org/pipermail/distutils-sig/2005-August/004954.html
msg164546 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-07-02 20:49
samtygier’s patch to implement the fallback-to-copy idea looks good.  Not sure if an automated test would be written, but samtygier says it was manually tested.
msg164554 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2012-07-02 21:32
> AFAICT using hard links only serves to save up a little time and disk
> space; it seems to me that always copying would solve one or two bugs
> at a small cost

I agree with this.
msg186994 - (view) Author: sorin (sorin) Date: 2013-04-15 14:25
Can we have this merged, it prevents us form using distutil, especially in a continuous integration environment where you do not have control over the build server.

See: https://drone.io/github.com/pycontribs/tendo/1
msg186996 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2013-04-15 14:39
I’ll get this in the next bugfix releases.
msg188970 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-05-11 23:41
Not blocking on this old bug.
msg208191 - (view) Author: Larry Hastings (larry) * (Python committer) Date: 2014-01-15 20:35
Has this been fixed?
msg208776 - (view) Author: Fabian Kochem (fkochem) Date: 2014-01-22 10:58
Éric said that he should be able to get to it soon.:
https://twitter.com/merwok_/status/425596183176704002
msg208792 - (view) Author: Fabian Kochem (fkochem) Date: 2014-01-22 12:23
A dirty hack is to include this line at the top of your setup.py:

del os.link
msg211180 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-02-13 22:07
Sorry for the lax definition of “soon”; I’m back from a busy January and will try to make some time for CPython contributions.
msg229298 - (view) Author: Matt Wright (Matt.Wright) Date: 2014-10-14 15:06
Here's another example of where this is a pain. An emerging workflow is using Docker for a Python environment. However, on OS X, its common to use boot2docker (a lightweight VM). With VirtualBox on OS X, its common to setup a shared folder between the host and boot2docker so one can mount volumes to Docker containers from the host to the container(s) on the VM. So when running something like `tox` or `python setup.py sdist` things fail because VirtualBox does not allow hard links on the shared folder filesystem. Changing this would be grand. Otherwise I have the, arguably, ugly `del os.link` in my setup.py.
msg230211 - (view) Author: Daniel Farrell (dfarrell07) * Date: 2014-10-29 15:03
> An emerging workflow is using Docker for a Python environment.

This applies to Vagrant environments as well (VirtualBox shared folders, tests fail).
msg230213 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-29 15:30
I agree this deserves fixing, and the patch looks basically ok. It would be nice to add a test, though (by mocking os.link()).

Éric, are you still here, or should someone else take over?
msg230215 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-10-29 15:33
I can’t make any commitment to core Python at this time.  I still read all email and can be available for guidance or questions about obscure parts of distutils.

(Ideally I would find someone willing to be mentored to take over maintenance.)
msg230242 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-29 20:25
Updated patch with tests.
msg230250 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-29 22:15
For the record, the only place where distutils seems to use hard-linking is distutils.command.sdist. The fallback on copying is safe there, since it's done manually if os.link doesn't exist.
msg230283 - (view) Author: Roundup Robot (python-dev) Date: 2014-10-30 18:39
New changeset d94d8789e924 by Antoine Pitrou in branch '3.4':
Issue #8876: distutils now falls back to copying files when hard linking doesn't work.
https://hg.python.org/cpython/rev/d94d8789e924

New changeset ce484e0840e3 by Antoine Pitrou in branch 'default':
Issue #8876: distutils now falls back to copying files when hard linking doesn't work.
https://hg.python.org/cpython/rev/ce484e0840e3
msg230285 - (view) Author: Roundup Robot (python-dev) Date: 2014-10-30 18:48
New changeset 263395345aa7 by Antoine Pitrou in branch '2.7':
Issue #8876: distutils now falls back to copying files when hard linking doesn't work.
https://hg.python.org/cpython/rev/263395345aa7
msg230286 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-10-30 18:54
This should now be fixed. Thanks for your patience :)
msg230290 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-10-30 19:14
Thanks for the patches folks!
History
Date User Action Args
2014-10-30 19:14:46eric.araujosetmessages: + msg230290
2014-10-30 18:54:39pitrousetstatus: open -> closed
resolution: fixed
messages: + msg230286

stage: patch review -> resolved
2014-10-30 18:48:58python-devsetmessages: + msg230285
2014-10-30 18:39:35python-devsetnosy: + python-dev
messages: + msg230283
2014-10-29 22:15:24pitrousetmessages: + msg230250
2014-10-29 20:25:28pitrousetfiles: + copy_file_link_fail.patch

messages: + msg230242
stage: test needed -> patch review
2014-10-29 20:09:47pitrousetassignee: eric.araujo ->
stage: patch review -> test needed
versions: + Python 3.5, - Python 3.3
2014-10-29 15:33:03eric.araujosetmessages: + msg230215
2014-10-29 15:30:25pitrousetmessages: + msg230213
2014-10-29 15:03:09dfarrell07setnosy: + dfarrell07
messages: + msg230211
2014-10-14 15:06:40Matt.Wrightsetnosy: + Matt.Wright
messages: + msg229298
2014-02-13 22:07:21eric.araujosetmessages: + msg211180
2014-01-22 12:23:48fkochemsetmessages: + msg208792
2014-01-22 10:58:40fkochemsetmessages: + msg208776
2014-01-15 20:35:18larrysetmessages: + msg208191
2014-01-12 21:26:50fkochemsetnosy: + fkochem
2013-05-11 23:41:51benjamin.petersonsetpriority: release blocker -> normal

messages: + msg188970
2013-04-15 14:39:13eric.araujosetpriority: normal -> release blocker
versions: + Python 3.4, - Python 3.2
nosy: + larry, benjamin.peterson, georg.brandl

messages: + msg186996

keywords: - needs review
2013-04-15 14:25:32sorinsetnosy: + sorin
messages: + msg186994
2012-07-02 21:32:05pitrousetmessages: + msg164554
2012-07-02 20:49:09eric.araujosetversions: - 3rd party
messages: + msg164546

components: - Distutils2
keywords: + needs review
stage: needs patch -> patch review
2012-07-02 20:44:58eric.araujosetmessages: + msg164545
2012-07-02 20:37:24eric.araujosetversions: + Python 3.3, - Python 3.1
nosy: + lemburg, pitrou, olliewalsh

messages: + msg164543

assignee: tarek -> eric.araujo
stage: needs patch
2010-09-30 01:36:28eric.araujosettype: behavior
versions: + 3rd party, Python 3.1, Python 2.7, Python 3.2, - Python 2.6
2010-06-02 14:43:48eric.araujosetnosy: + eric.araujo
messages: + msg106886
2010-06-02 14:36:21samtygiersetfiles: + distutil-hardlink-trunk.diff
keywords: + patch
messages: + msg106885
2010-06-02 13:40:59eric.araujosetnosy: tarek, samtygier
components: + Distutils2
2010-06-02 13:38:35samtygiercreate