New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Have os.unlink remove junction points #62514
Comments
os.unlink currently raises a WindowsError (Access Denied) if I attempt to unlink an NTFS junction point. It looks trivial to allow Py_DeleteFileW [1] to remove junction points as well as proper symbolic links, as far as I can tell. For example, the ntfslink-python library [2] only checks if both FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_REPARSE_POINT are set. RemoveDirectoryW is documented to handle junction points transparently, so it should just be a matter of passing the path on if it's a junction point or a symbolic link. My motivation for this is that I have used external tools to create junction points, and am now switching to symbolic links. When deleting a directory, I need to do:
which is a little funky. It seems like os.unlink semantics work just as well for junction points, even if they can't be created with os.symlink. Love it/hate it? [1] http://hg.python.org/cpython/file/44f455e6163d/Modules/posixmodule.c#l4105 |
Also, I believe the reason os.unlink raises "access denied" is because a junction point does not currently qualify as a directory && link, so its path is passed directly to DeleteFileW, which in turn refuses to delete a directory. |
This comment outlines how to tell junction points from other mount points: This should port straight into Py_DeleteFileW. Would anyone be interested in a patch? |
What is "an NTFS junction point"? Is it possible to delete it in cmd.exe with the del command? |
Victor, Wikipedia has a readable explanation: http://en.wikipedia.org/wiki/NTFS_junction_point I haven't used them much. From cmd.exe, I've been able to delete them, not with "del" but with "rmdir". You can create one from cmd.exe with the "mklink" command. |
Victor, Junction points are like links between directories only. They've been around since the NTFS that came with Windows 2000, but integration with OS tools has been generally poor (e.g. Explorer wouldn't see the difference between a junction point and a regular folder.) As of Windows 7, this works better. |
I am not sure if this is a bug or enhancement. It is a moot point until there is a patch to apply. A patch would need a test that fails now and passes with the patch. From the Wikipedia article, it appears that a test using mklink /J would not run on XP and would have to be skipped. I would not expect an XP buildbot to necessarily have the Server 2003 Resource Kit needed for an XP test. Does _delete_junction_point(link_path) work on XP or should the feature be restricted to Vista+? |
_delete_junction_point currently shells out to a command-line tool, junction.exe, from SysInternals. That ran fine on XP. As I understand it, RemoveDirectoryW on XP also takes care of junction points, but I'll find a machine to verify. |
Attached is a patch that considers directory symlinks and junction points equivalent. I'm struggling with the test -- would it be acceptable to only run this test on platforms that have mklink /j (Windows Vista and higher)? I've looked at programmatic means of creating junction points, but it involves enough Win32 interop to make it a candidate for a module all by itself (it's REALLY messy.) Any ideas? Thanks! |
I'll try to pick this one up over the next few days. Feel free to ping me if it drops into silence! |
Gentle ping. |
Just picking this up. Considering testing... My current proposal is to add junction point support to _winapi, initially for the sole purpose of testing this change, but with a view to possibly supporting it formally via the os module. Any better ideas? |
I didn't know about _winapi; looks like a good place! It looks like it exposes the Windows API pretty faithfully, but the junction point stuff feels like it would be better represented as a simple _winapi.CreateJunctionPoint(source, target) rather than attempting to expose DeviceIoControl and associated structures. I'll try to cook up a patch and we can argue about details based on that :-) Thanks! |
Sounds like a decent plan to me. Good luck with the buffer sizing! |
I really needed the well-wishing with regard to buffer sizing :-) Here's a patch for a couple of fronts:
I pulled the definition of _REPARSE_DATA_BUFFER out into a new header called winreparse.h. I'd appreciate critical review of _winapi.CreateJunction to make sure I haven't missed anything. I'm not familiar with the Python/C interop idioms, so I might have missed something wrt arguments/return values handling. Happy new year! |
I'll have a look at this in a week or so when I'm back on-line. |
Thanks! There's another thing I would appreciate having somebody else test: creating and removing junctions in a non-elevated prompt. I haven't been able to, my IT department has trouble understanding the value of least-privilege. |
ping |
1 similar comment
ping |
All tests pass on 3.5 and in an unelevated prompt. I'll have a closer look at the code tomorrow. |
New changeset 17df50df62c7 by Tim Golden in branch 'default': |
New changeset 4b97092aa4bd by Tim Golden in branch 'default': |
Backed out the commits after all the Windows buildbots broke. Need to look further. (No problems on a Win7 or Ubuntu build here). |
Thanks for pushing this forward! Do you have links to the failing bots I could take a look at? |
Thanks! At first I suspected 32 vs 64 bit, but the failing bots cover both... One thing that stands out to me as risky is the memcmp() against "\\??\\", which could overrun a short src_path buffer. But I don't think that would fail here. I must have made some mistake with the REPARSE_DATA_BUFFER, but I can't see anything off hand. What are our debugging options? |
I'm just pinging #python-dev to see if there's a way to request a buildbot build from a specific server-side clone. Meanwhile, though, I definitely introduced a change into your code which I thought I had reverted, but clearly hadn't! The code, as committed, used PyMem_RawAlloc in place of the calloc() call you had, but didn't replace the later free() by its PyMem counterpart. If I don't get any joy with the clone-specific buildbot question, I'll just rebuild from your original patch, re-commit, and watch the buildbots. |
Aha, that might cause trouble. I think you should add a memset() to sero out the newly allocated buffer also, I think I may have used calloc to be able to assume it was initialized with zeros. |
Sorry, that wasn't clear. I mean if you change allocator _from_ calloc, make sure the buffer is zeroed out after allocation. |
By the way, is PyMem_RawMalloc/PyMem_RawFree preferred for memory allocation across the board? If so, I can just prepare a new patch for you with that changed, zero-initialization in place and the prefix-overrun fixed. I might get to it tonight. |
Yes, now that the custom allocator / tracing stuff is in place: Also, since the setup of the reparse header is such an underdocumented (BTW I'm not convinced that the PyMem change was the problem since the |
I'll try. It might turn into a novel.
I think it might be, there was a message in the log that DeviceIoControl failed: stty: standard input: Inappropriate ioctl for device That could be attributed to garbage in the buffer. I'll be back with a revised patch, and we can work from there. Thanks for your help! |
Here's a new attempt, please let me know if this works out better. Changes:
Hope this works out better. I already have ideas for improvements, but I think we can try to get this in place first. |
New changeset 5c1c14ff1f13 by Tim Golden in branch 'default': |
Just skimming, I noticed something about replacing calloc() with PyMem_RawAlloc; note that there is now PyMem_Calloc or PyMem_RawCalloc that you should be able to use if you prefer. See bpo-21233. |
Thanks, Zach. I was aware that calloc was in the air, but I wasn't sure |
New changeset e791a36ab6a2 by Tim Golden in branch 'default': |
Buildbots seem happy. Thanks very much for the patches! |
For a junction reparse point, Sysinternals junction.exe shows the "Print Name" and "Substitute Name" are the same and it's not an NT \?? path (i.e. \DosDevices, i.e. \Global??). OTOH, the substitute name does use an NT DosDevices path when I use cmd's mklink or os.symlink to create a directory symbolic link to an absolute path. I modified the patch in a test program to see whether it's really necessary to include the DosDevices path in the substitute name. FYI, it seems to work fine without it, but it also doesn't seem to hurt to include it. Sysinternals Junction |
Thanks for the research, eryksun. As long as it doesn't hurt let's leave |
Thanks for helping me land this! eryksun: interesting, thanks! I seem to remember having problems without the \??\ prefix, but it could have been something else causing it (filling the buffer to DeviceIoControl's satisfaction was... challenging.) I have some ideas for small improvements, and I could try to fold the removal of \??\ into that. Do I just open a new enhancement issue? |
Yes, new issue. |
Nevermind, strike "seems to work"; it doesn't work without the \?? prefix. I stupidly assumed the DeviceIoControl call would validate the substitute name. Of course it doesn't; it happily creates a broken junction. Opening the junction with CreateFile fails with either ERROR_INVALID_REPARSE_DATA or ERROR_CANT_RESOLVE_FILENAME. Sorry. |
Thanks, eryksun: failed experiments are still useful data for future |
For some read Sysinternals junction utility doesn't show the raw substitute name. Microsoft's fsutil shows the actual reparse data:
fsutil reparsepoint: |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: