os.unlink currently raises a WindowsError (Access Denied) if I attempt to unlink an NTFS junction point.

It looks trivial to allow Py_DeleteFileW [1] to remove junction points as well as proper symbolic links, as far as I can tell.

For example, the ntfslink-python library [2] only checks if both FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_REPARSE_POINT are set.

RemoveDirectoryW is documented to handle junction points transparently, so it should just be a matter of passing the path on if it's a junction point or a symbolic link.

My motivation for this is that I have used external tools to create junction points, and am now switching to symbolic links. When deleting a directory, I need to do:

    try:
        os.unlink(link_path)
    except WindowsError as detail:
        # BACKWARD COMPATIBILITY HACK
        if detail.winerror == 5:
            _delete_junction_point(link_path)
        else:
            raise

which is a little funky. It seems like os.unlink semantics work just as well for junction points, even if they can't be created with os.symlink.

Love it/hate it?

[1] http://hg.python.org/cpython/file/44f455e6163d/Modules/posixmodule.c#l4105
[2] https://github.com/Juntalis/ntfslink-python/blob/2f6ff903f9b22942de8aa93a32a3d817124f359e/ntfslink/internals/__init__.py#L32

Also, I believe the reason os.unlink raises "access denied" is because a junction point does not currently qualify as a directory && link, so its path is passed directly to DeleteFileW, which in turn refuses to delete a directory.

This comment outlines how to tell junction points from other mount points:
http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista?msg=3651130#xx3651130xx

This should port straight into Py_DeleteFileW.

Would anyone be interested in a patch?

What is "an NTFS junction point"? Is it possible to delete it in cmd.exe with the del command?

Victor, Wikipedia has a readable explanation:

http://en.wikipedia.org/wiki/NTFS_junction_point

I haven't used them much.  From cmd.exe, I've been able to delete them, not with "del" but with "rmdir".  You can create one from cmd.exe with the "mklink" command.

Victor,

Junction points are like links between directories only. They've been around since the NTFS that came with Windows 2000, but integration with OS tools has been generally poor (e.g. Explorer wouldn't see the difference between a junction point and a regular folder.) As of Windows 7, this works better.

I am not sure if this is a bug or enhancement. It is a moot point until there is a patch to apply.

A patch would need a test that fails now and passes with the patch. From the Wikipedia article, it appears that a test using mklink /J would not run on XP and would have to be skipped. I would not expect an XP buildbot to necessarily have the Server 2003 Resource Kit needed for an XP test.

Does _delete_junction_point(link_path) work on XP or should the feature be restricted to Vista+?

_delete_junction_point currently shells out to a command-line tool, junction.exe, from SysInternals. That ran fine on XP.

As I understand it, RemoveDirectoryW on XP also takes care of junction points, but I'll find a machine to verify.

Attached is a patch that considers directory symlinks and junction points equivalent.

I'm struggling with the test -- would it be acceptable to only run this test on platforms that have mklink /j (Windows Vista and higher)?

I've looked at programmatic means of creating junction points, but it involves enough Win32 interop to make it a candidate for a module all by itself (it's REALLY messy.)

Any ideas?

Thanks!

I'll try to pick this one up over the next few days. Feel free to ping me if it drops into silence!

Gentle ping.

Just picking this up. Considering testing... My current proposal is to add junction point support to _winapi, initially for the sole purpose of testing this change, but with a view to possibly supporting it formally via the os module. Any better ideas?

I didn't know about _winapi; looks like a good place!

It looks like it exposes the Windows API pretty faithfully, but the junction point stuff feels like it would be better represented as a simple _winapi.CreateJunctionPoint(source, target) rather than attempting to expose DeviceIoControl and associated structures.

I'll try to cook up a patch and we can argue about details based on that :-)

Thanks!

Sounds like a decent plan to me. Good luck with the buffer sizing!

I really needed the well-wishing with regard to buffer sizing :-)

Here's a patch for a couple of fronts:
- Teach os.unlink about junction points
- Introduce _winapi.CreateJunction
- Introduce a new test suite in test_os.py for junction points

I pulled the definition of _REPARSE_DATA_BUFFER out into a new header called winreparse.h.

I'd appreciate critical review of _winapi.CreateJunction to make sure I haven't missed anything. I'm not familiar with the Python/C interop idioms, so I might have missed something wrt arguments/return values handling.

Happy new year!

I'll have a look at this in a week or so when I'm back on-line.

Thanks!

There's another thing I would appreciate having somebody else test: creating and removing junctions in a non-elevated prompt. I haven't been able to, my IT department has trouble understanding the value of least-privilege.

ping

ping

All tests pass on 3.5 and in an unelevated prompt. I'll have a closer look at the code tomorrow.

New changeset 17df50df62c7 by Tim Golden in branch 'default':
Issue #18314 os.unlink will now remove junction points on Windows. Patch by Kim Gräsman.
http://hg.python.org/cpython/rev/17df50df62c7

New changeset 4b97092aa4bd by Tim Golden in branch 'default':
Issue #18314 Add NEWS item.
http://hg.python.org/cpython/rev/4b97092aa4bd

Backed out the commits after all the Windows buildbots broke. Need to look further. (No problems on a Win7 or Ubuntu build here).

Thanks for pushing this forward! Do you have links to the failing bots I could take a look at?

Here are a couple:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4423

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/8288

Thanks!

At first I suspected 32 vs 64 bit, but the failing bots cover both...

One thing that stands out to me as risky is the memcmp() against "\\??\\", which could overrun a short src_path buffer. But I don't think that would fail here.

I must have made some mistake with the REPARSE_DATA_BUFFER, but I can't see anything off hand.

What are our debugging options?

I'm just pinging #python-dev to see if there's a way to request a buildbot build from a specific server-side clone. 

Meanwhile, though, I definitely introduced a change into your code which I thought I had reverted, but clearly hadn't! The code, as committed, used PyMem_RawAlloc in place of the calloc() call you had, but didn't replace the later free() by its PyMem counterpart.

If I don't get any joy with the clone-specific buildbot question, I'll just rebuild from your original patch, re-commit, and watch the buildbots.

Aha, that might cause trouble.

I think you should add a memset() to sero out the newly allocated buffer also, I think I may have used calloc to be able to assume it was initialized with zeros.

Sorry, that wasn't clear. I mean if you change allocator _from_ calloc, make sure the buffer is zeroed out after allocation.

By the way, is PyMem_RawMalloc/PyMem_RawFree preferred for memory allocation across the board?

If so, I can just prepare a new patch for you with that changed, zero-initialization in place and the prefix-overrun fixed. I might get to it tonight.

Yes, now that the custom allocator / tracing stuff is in place: 
otherwise there's no way for custom allocation or tracing to occur. 
Please go ahead and rework the patch when you have the time.

Also, since the setup of the reparse header is such an underdocumented 
nightmare, please add as much commentary as possible around the choice 
of allocations & offsets. I was reviewing your code with an eye on the 
various MSDN pages and examples and I still wasn't always able to follow 
your choices.

(BTW I'm not convinced that the PyMem change was the problem since the 
PyMem_Raw* functions simply hand off to malloc/free unless there's a 
custom allocator. Unless of course the missing memset-0 was significant 
here).

> Also, since the setup of the reparse header is such an underdocumented
> nightmare, please add as much commentary as possible around the choice 
> of allocations & offsets.

I'll try. It might turn into a novel.

> (BTW I'm not convinced that the PyMem change was the problem since 
> the PyMem_Raw* functions simply hand off to malloc/free unless
> there's a custom allocator.
> Unless of course the missing memset-0 was significant here).

I think it might be, there was a message in the log that DeviceIoControl failed:

  stty: standard input: Inappropriate ioctl for device

That could be attributed to garbage in the buffer.

I'll be back with a revised patch, and we can work from there. Thanks for your help!

Here's a new attempt, please let me know if this works out better.

Changes:
- Switched to CRT string functions (wcsncmp, wcscpy) instead of Windows lstrxxxW. There was no lstrncmpW.
- Switched to PyMem_Raw(Malloc|Free) and added explicit memset after allocation
- Better error handling (check arguments for NULL, check memory allocation)
- Fix possible overrun when checking if src_path starts with "\??\"
- Extensive commentary to describe the buffer sizing

Hope this works out better. I already have ideas for improvements, but I think we can try to get this in place first.

New changeset 5c1c14ff1f13 by Tim Golden in branch 'default':
Issue18314 Allow unlink to remove junctions. Includes support for creating junctions. Patch by Kim Gräsman
http://hg.python.org/cpython/rev/5c1c14ff1f13

Just skimming, I noticed something about replacing calloc() with PyMem_RawAlloc; note that there is now PyMem_Calloc or PyMem_RawCalloc that you should be able to use if you prefer.  See #21233.

Thanks, Zach. I was aware that calloc was in the air, but I wasn't sure 
if it had been committed and I'd delayed on this enough so I thought I'd 
push for now. We can always revisit.

New changeset e791a36ab6a2 by Tim Golden in branch 'default':
Issue18314 ACKS & NEWS
http://hg.python.org/cpython/rev/e791a36ab6a2

Buildbots seem happy. Thanks very much for the patches!

For a junction reparse point, Sysinternals junction.exe shows the "Print Name" and "Substitute Name" are the same and it's not an NT \?? path (i.e. \DosDevices, i.e. \Global??). OTOH, the substitute name does use an NT DosDevices path when I use cmd's mklink or os.symlink to create a directory symbolic link to an absolute path. 

I modified the patch in a test program to see whether it's really necessary to include the DosDevices path in the substitute name. FYI, it seems to work fine without it, but it also doesn't seem to hurt to include it.

Sysinternals Junction
http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx

Thanks for the research, eryksun. As long as it doesn't hurt let's leave 
it as is for now.

Thanks for helping me land this!

eryksun: interesting, thanks! I seem to remember having problems without the \??\ prefix, but it could have been something else causing it (filling the buffer to DeviceIoControl's satisfaction was... challenging.)

I have some ideas for small improvements, and I could try to fold the removal of \??\ into that. Do I just open a new enhancement issue?

Yes, new issue.

Nevermind, strike "seems to work"; it doesn't work without the \?? prefix. I stupidly assumed the DeviceIoControl call would validate the substitute name. Of course it doesn't; it happily creates a broken junction. Opening the junction with CreateFile fails with either ERROR_INVALID_REPARSE_DATA or ERROR_CANT_RESOLVE_FILENAME. Sorry.

Thanks, eryksun: failed experiments are still useful data for future
reference!

For some read Sysinternals junction utility doesn't show the raw substitute name. Microsoft's fsutil shows the actual reparse data:

    C:\>mklink /J Python34 "C:\Program Files\Python34"
    Junction created for Python34 <<===>> C:\Program Files\Python34

    C:\>fsutil reparsepoint query Python34
    Reparse Tag Value : 0xa0000003
    Tag value: Microsoft
    Tag value: Name Surrogate
    Tag value: Mount Point
    Substitue Name offset: 0
    Substitue Name length: 58
    Print Name offset:     60
    Print Name Length:     50
    Substitute Name:       \??\C:\Program Files\Python34
    Print Name:            C:\Program Files\Python34

    Reparse Data Length: 0x00000078
    Reparse Data:
    0000:  00 00 3a 00 3c 00 32 00  5c 00 3f 00 3f 00 5c 00  ..:.<.2.\.?.?.\.
    0010:  43 00 3a 00 5c 00 50 00  72 00 6f 00 67 00 72 00  C.:.\.P.r.o.g.r.
    0020:  61 00 6d 00 20 00 46 00  69 00 6c 00 65 00 73 00  a.m. .F.i.l.e.s.
    0030:  5c 00 50 00 79 00 74 00  68 00 6f 00 6e 00 33 00  \.P.y.t.h.o.n.3.
    0040:  34 00 00 00 43 00 3a 00  5c 00 50 00 72 00 6f 00  4...C.:.\.P.r.o.
    0050:  67 00 72 00 61 00 6d 00  20 00 46 00 69 00 6c 00  g.r.a.m. .F.i.l.
    0060:  65 00 73 00 5c 00 50 00  79 00 74 00 68 00 6f 00  e.s.\.P.y.t.h.o.
    0070:  6e 00 33 00 34 00 00 00                           n.3.4...

fsutil reparsepoint:
http://technet.microsoft.com/en-us/library/cc785451.aspx