New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
distutils should not assume that hardlinks will work #53122
Comments
distutils will currently try to use hardlinks if os has a 'link' attribute, however sometimes os.link() will fail, for example the filesystem may not support it (see attached traceback). in commands/sdist.py in make_release_tree() there is the test: 'link' is then passed to copy_file() in file_util.py, which trusts that if link == 'hard', then hardlinking will work. there has been discussion in the past, but i dont think it has been fixed |
here is a patch against http://svn.python.org/projects/python/trunk it moves the return statements into the individual file copying sections, and takes the final call to _copy_file_contents() out of the else. this allows an error in hardlinking to fall through by passing the exception. i have only tested on Linux. when running it still prints out i have a patch that applies to 2.6 as well, is there much chance of that being accepted? |
Distutils is a special case: Many third-party code relies on its undocumented quirks and bugs, so it’s basically frozen. Some non-disruptive bugfixes are accepted, in which case the normal Python rules apply (e.g., no new features in 2.7 which is in beta). Work has been moved to Distutils2, which breaks compatibility in order to fix things and add needed features. If Tarek (maintainer of both packages) agrees this can go into Distutils, Benjamin Peterson (release manager) will say whether he agrees or not. Patient a while for Tarek to see this bug report, and thanks for your work :) |
AFAICT using hard links only serves to save up a little time and disk space; it seems to me that always copying would solve one or two bugs at a small cost (not so small for large projects, I don’t know). Could this impact setup scripts? Maybe if os.link were monkey-patched and expected to be called. I could ask on distutils-sig. In bpo-15205 Ollie noted this: “distutils2 appears to always copy instead of hardlinking resolving all of these issues”. That would be because d2 uses shutil.copytree and distutils.util.copy_file (yes, a function from distutils1 because of some Windows issue :(). |
Of course the conservative fix would be to try linking as of now and do a copy when an exception is caught, as was proposed. FTR python.org link to the thread mentioned: http://mail.python.org/pipermail/distutils-sig/2005-August/004954.html |
samtygier’s patch to implement the fallback-to-copy idea looks good. Not sure if an automated test would be written, but samtygier says it was manually tested. |
I agree with this. |
Can we have this merged, it prevents us form using distutil, especially in a continuous integration environment where you do not have control over the build server. |
I’ll get this in the next bugfix releases. |
Not blocking on this old bug. |
Has this been fixed? |
Éric said that he should be able to get to it soon.: |
A dirty hack is to include this line at the top of your setup.py: del os.link |
Sorry for the lax definition of “soon”; I’m back from a busy January and will try to make some time for CPython contributions. |
Here's another example of where this is a pain. An emerging workflow is using Docker for a Python environment. However, on OS X, its common to use boot2docker (a lightweight VM). With VirtualBox on OS X, its common to setup a shared folder between the host and boot2docker so one can mount volumes to Docker containers from the host to the container(s) on the VM. So when running something like |
This applies to Vagrant environments as well (VirtualBox shared folders, tests fail). |
I agree this deserves fixing, and the patch looks basically ok. It would be nice to add a test, though (by mocking os.link()). Éric, are you still here, or should someone else take over? |
I can’t make any commitment to core Python at this time. I still read all email and can be available for guidance or questions about obscure parts of distutils. (Ideally I would find someone willing to be mentored to take over maintenance.) |
Updated patch with tests. |
For the record, the only place where distutils seems to use hard-linking is distutils.command.sdist. The fallback on copying is safe there, since it's done manually if os.link doesn't exist. |
New changeset d94d8789e924 by Antoine Pitrou in branch '3.4': New changeset ce484e0840e3 by Antoine Pitrou in branch 'default': |
New changeset 263395345aa7 by Antoine Pitrou in branch '2.7': |
This should now be fixed. Thanks for your patience :) |
Thanks for the patches folks! |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: