classification
Title: Have os.unlink remove junction points
Type: enhancement Stage: resolved
Components: Windows Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: tim.golden Nosy List: Kim.Gräsman, brian.curtin, eryksun, python-dev, terry.reedy, tim.golden, tim.peters, zach.ware
Priority: normal Keywords: patch

Created on 2013-06-27 12:15 by Kim.Gräsman, last changed 2014-05-07 00:33 by eryksun. This issue is now closed.

Files
File name Uploaded Description Edit
unlink_junction.patch Kim.Gräsman, 2013-09-29 20:34 review
unlink-junctions.patch Kim.Gräsman, 2013-12-30 15:51 review
unlink_junctions_r2.patch Kim.Gräsman, 2014-04-29 18:58 review
Messages (45)
msg191945 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-06-27 12:15
os.unlink currently raises a WindowsError (Access Denied) if I attempt to unlink an NTFS junction point.

It looks trivial to allow Py_DeleteFileW [1] to remove junction points as well as proper symbolic links, as far as I can tell.

For example, the ntfslink-python library [2] only checks if both FILE_ATTRIBUTE_DIRECTORY and FILE_ATTRIBUTE_REPARSE_POINT are set.

RemoveDirectoryW is documented to handle junction points transparently, so it should just be a matter of passing the path on if it's a junction point or a symbolic link.

My motivation for this is that I have used external tools to create junction points, and am now switching to symbolic links. When deleting a directory, I need to do:

    try:
        os.unlink(link_path)
    except WindowsError as detail:
        # BACKWARD COMPATIBILITY HACK
        if detail.winerror == 5:
            _delete_junction_point(link_path)
        else:
            raise

which is a little funky. It seems like os.unlink semantics work just as well for junction points, even if they can't be created with os.symlink.

Love it/hate it?

[1] http://hg.python.org/cpython/file/44f455e6163d/Modules/posixmodule.c#l4105
[2] https://github.com/Juntalis/ntfslink-python/blob/2f6ff903f9b22942de8aa93a32a3d817124f359e/ntfslink/internals/__init__.py#L32
msg191946 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-06-27 12:21
Also, I believe the reason os.unlink raises "access denied" is because a junction point does not currently qualify as a directory && link, so its path is passed directly to DeleteFileW, which in turn refuses to delete a directory.
msg192177 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-07-02 07:24
This comment outlines how to tell junction points from other mount points:
http://www.codeproject.com/Articles/21202/Reparse-Points-in-Vista?msg=3651130#xx3651130xx

This should port straight into Py_DeleteFileW.

Would anyone be interested in a patch?
msg195114 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-08-14 00:57
What is "an NTFS junction point"? Is it possible to delete it in cmd.exe with the del command?
msg195121 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2013-08-14 03:48
Victor, Wikipedia has a readable explanation:

http://en.wikipedia.org/wiki/NTFS_junction_point

I haven't used them much.  From cmd.exe, I've been able to delete them, not with "del" but with "rmdir".  You can create one from cmd.exe with the "mklink" command.
msg195124 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-08-14 04:40
Victor,

Junction points are like links between directories only. They've been around since the NTFS that came with Windows 2000, but integration with OS tools has been generally poor (e.g. Explorer wouldn't see the difference between a junction point and a regular folder.) As of Windows 7, this works better.
msg198416 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-09-25 20:52
I am not sure if this is a bug or enhancement. It is a moot point until there is a patch to apply.

A patch would need a test that fails now and passes with the patch. From the Wikipedia article, it appears that a test using mklink /J would not run on XP and would have to be skipped. I would not expect an XP buildbot to necessarily have the Server 2003 Resource Kit needed for an XP test.

Does _delete_junction_point(link_path) work on XP or should the feature be restricted to Vista+?
msg198500 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-09-27 19:46
_delete_junction_point currently shells out to a command-line tool, junction.exe, from SysInternals. That ran fine on XP.

As I understand it, RemoveDirectoryW on XP also takes care of junction points, but I'll find a machine to verify.
msg198647 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-09-29 20:34
Attached is a patch that considers directory symlinks and junction points equivalent.

I'm struggling with the test -- would it be acceptable to only run this test on platforms that have mklink /j (Windows Vista and higher)?

I've looked at programmatic means of creating junction points, but it involves enough Win32 interop to make it a candidate for a module all by itself (it's REALLY messy.)

Any ideas?

Thanks!
msg198649 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-09-29 20:59
I'll try to pick this one up over the next few days. Feel free to ping me if it drops into silence!
msg200058 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-10-16 12:31
Gentle ping.
msg201018 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-10-23 11:42
Just picking this up. Considering testing... My current proposal is to add junction point support to _winapi, initially for the sole purpose of testing this change, but with a view to possibly supporting it formally via the os module. Any better ideas?
msg201054 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-10-23 18:55
I didn't know about _winapi; looks like a good place!

It looks like it exposes the Windows API pretty faithfully, but the junction point stuff feels like it would be better represented as a simple _winapi.CreateJunctionPoint(source, target) rather than attempting to expose DeviceIoControl and associated structures.

I'll try to cook up a patch and we can argue about details based on that :-)

Thanks!
msg201055 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-10-23 18:56
Sounds like a decent plan to me. Good luck with the buffer sizing!
msg207097 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2013-12-30 15:51
I really needed the well-wishing with regard to buffer sizing :-)

Here's a patch for a couple of fronts:
- Teach os.unlink about junction points
- Introduce _winapi.CreateJunction
- Introduce a new test suite in test_os.py for junction points

I pulled the definition of _REPARSE_DATA_BUFFER out into a new header called winreparse.h.

I'd appreciate critical review of _winapi.CreateJunction to make sure I haven't missed anything. I'm not familiar with the Python/C interop idioms, so I might have missed something wrt arguments/return values handling.

Happy new year!
msg207157 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-01-02 10:45
I'll have a look at this in a week or so when I'm back on-line.
msg208500 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-01-19 21:25
Thanks!

There's another thing I would appreciate having somebody else test: creating and removing junctions in a non-elevated prompt. I haven't been able to, my IT department has trouble understanding the value of least-privilege.
msg211008 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-02-11 19:58
ping
msg213278 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-03-12 19:10
ping
msg216579 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-04-16 20:16
All tests pass on 3.5 and in an unelevated prompt. I'll have a closer look at the code tomorrow.
msg217287 - (view) Author: Roundup Robot (python-dev) Date: 2014-04-27 17:00
New changeset 17df50df62c7 by Tim Golden in branch 'default':
Issue #18314 os.unlink will now remove junction points on Windows. Patch by Kim Gräsman.
http://hg.python.org/cpython/rev/17df50df62c7
msg217288 - (view) Author: Roundup Robot (python-dev) Date: 2014-04-27 17:03
New changeset 4b97092aa4bd by Tim Golden in branch 'default':
Issue #18314 Add NEWS item.
http://hg.python.org/cpython/rev/4b97092aa4bd
msg217293 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-04-27 17:38
Backed out the commits after all the Windows buildbots broke. Need to look further. (No problems on a Win7 or Ubuntu build here).
msg217311 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-27 19:24
Thanks for pushing this forward! Do you have links to the failing bots I could take a look at?
msg217312 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-04-27 19:27
Here are a couple:

http://buildbot.python.org/all/builders/AMD64%20Windows7%20SP1%203.x/builds/4423

http://buildbot.python.org/all/builders/x86%20Windows7%203.x/builds/8288
msg217318 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-27 21:04
Thanks!

At first I suspected 32 vs 64 bit, but the failing bots cover both...

One thing that stands out to me as risky is the memcmp() against "\\??\\", which could overrun a short src_path buffer. But I don't think that would fail here.

I must have made some mistake with the REPARSE_DATA_BUFFER, but I can't see anything off hand.

What are our debugging options?
msg217404 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-04-28 19:28
I'm just pinging #python-dev to see if there's a way to request a buildbot build from a specific server-side clone. 

Meanwhile, though, I definitely introduced a change into your code which I thought I had reverted, but clearly hadn't! The code, as committed, used PyMem_RawAlloc in place of the calloc() call you had, but didn't replace the later free() by its PyMem counterpart.

If I don't get any joy with the clone-specific buildbot question, I'll just rebuild from your original patch, re-commit, and watch the buildbots.
msg217463 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-29 04:01
Aha, that might cause trouble.

I think you should add a memset() to sero out the newly allocated buffer also, I think I may have used calloc to be able to assume it was initialized with zeros.
msg217464 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-29 04:26
Sorry, that wasn't clear. I mean if you change allocator _from_ calloc, make sure the buffer is zeroed out after allocation.
msg217471 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-29 05:55
By the way, is PyMem_RawMalloc/PyMem_RawFree preferred for memory allocation across the board?

If so, I can just prepare a new patch for you with that changed, zero-initialization in place and the prefix-overrun fixed. I might get to it tonight.
msg217479 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-04-29 07:48
Yes, now that the custom allocator / tracing stuff is in place: 
otherwise there's no way for custom allocation or tracing to occur. 
Please go ahead and rework the patch when you have the time.

Also, since the setup of the reparse header is such an underdocumented 
nightmare, please add as much commentary as possible around the choice 
of allocations & offsets. I was reviewing your code with an eye on the 
various MSDN pages and examples and I still wasn't always able to follow 
your choices.

(BTW I'm not convinced that the PyMem change was the problem since the 
PyMem_Raw* functions simply hand off to malloc/free unless there's a 
custom allocator. Unless of course the missing memset-0 was significant 
here).
msg217488 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-29 08:25
> Also, since the setup of the reparse header is such an underdocumented
> nightmare, please add as much commentary as possible around the choice 
> of allocations & offsets.

I'll try. It might turn into a novel.

> (BTW I'm not convinced that the PyMem change was the problem since 
> the PyMem_Raw* functions simply hand off to malloc/free unless
> there's a custom allocator.
> Unless of course the missing memset-0 was significant here).

I think it might be, there was a message in the log that DeviceIoControl failed:

  stty: standard input: Inappropriate ioctl for device

That could be attributed to garbage in the buffer.

I'll be back with a revised patch, and we can work from there. Thanks for your help!
msg217538 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-04-29 18:58
Here's a new attempt, please let me know if this works out better.

Changes:
- Switched to CRT string functions (wcsncmp, wcscpy) instead of Windows lstrxxxW. There was no lstrncmpW.
- Switched to PyMem_Raw(Malloc|Free) and added explicit memset after allocation
- Better error handling (check arguments for NULL, check memory allocation)
- Fix possible overrun when checking if src_path starts with "\??\"
- Extensive commentary to describe the buffer sizing

Hope this works out better. I already have ideas for improvements, but I think we can try to get this in place first.
msg217940 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-05 18:47
New changeset 5c1c14ff1f13 by Tim Golden in branch 'default':
Issue18314 Allow unlink to remove junctions. Includes support for creating junctions. Patch by Kim Gräsman
http://hg.python.org/cpython/rev/5c1c14ff1f13
msg217941 - (view) Author: Zachary Ware (zach.ware) * (Python committer) Date: 2014-05-05 18:59
Just skimming, I noticed something about replacing calloc() with PyMem_RawAlloc; note that there is now PyMem_Calloc or PyMem_RawCalloc that you should be able to use if you prefer.  See #21233.
msg217943 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-05-05 19:19
Thanks, Zach. I was aware that calloc was in the air, but I wasn't sure 
if it had been committed and I'd delayed on this enough so I thought I'd 
push for now. We can always revisit.
msg217948 - (view) Author: Roundup Robot (python-dev) Date: 2014-05-05 20:00
New changeset e791a36ab6a2 by Tim Golden in branch 'default':
Issue18314 ACKS & NEWS
http://hg.python.org/cpython/rev/e791a36ab6a2
msg217949 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-05-05 20:01
Buildbots seem happy. Thanks very much for the patches!
msg217953 - (view) Author: Eryk Sun (eryksun) * Date: 2014-05-05 20:20
For a junction reparse point, Sysinternals junction.exe shows the "Print Name" and "Substitute Name" are the same and it's not an NT \?? path (i.e. \DosDevices, i.e. \Global??). OTOH, the substitute name does use an NT DosDevices path when I use cmd's mklink or os.symlink to create a directory symbolic link to an absolute path. 

I modified the patch in a test program to see whether it's really necessary to include the DosDevices path in the substitute name. FYI, it seems to work fine without it, but it also doesn't seem to hurt to include it.

Sysinternals Junction
http://technet.microsoft.com/en-us/sysinternals/bb896768.aspx
msg217955 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-05-05 20:58
Thanks for the research, eryksun. As long as it doesn't hurt let's leave 
it as is for now.
msg217966 - (view) Author: Kim Gräsman (Kim.Gräsman) * Date: 2014-05-06 04:02
Thanks for helping me land this!

eryksun: interesting, thanks! I seem to remember having problems without the \??\ prefix, but it could have been something else causing it (filling the buffer to DeviceIoControl's satisfaction was... challenging.)

I have some ideas for small improvements, and I could try to fold the removal of \??\ into that. Do I just open a new enhancement issue?
msg217969 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2014-05-06 06:08
Yes, new issue.
msg217975 - (view) Author: Eryk Sun (eryksun) * Date: 2014-05-06 11:16
Nevermind, strike "seems to work"; it doesn't work without the \?? prefix. I stupidly assumed the DeviceIoControl call would validate the substitute name. Of course it doesn't; it happily creates a broken junction. Opening the junction with CreateFile fails with either ERROR_INVALID_REPARSE_DATA or ERROR_CANT_RESOLVE_FILENAME. Sorry.
msg217976 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2014-05-06 11:24
Thanks, eryksun: failed experiments are still useful data for future
reference!
msg218030 - (view) Author: Eryk Sun (eryksun) * Date: 2014-05-07 00:33
For some read Sysinternals junction utility doesn't show the raw substitute name. Microsoft's fsutil shows the actual reparse data:

    C:\>mklink /J Python34 "C:\Program Files\Python34"
    Junction created for Python34 <<===>> C:\Program Files\Python34

    C:\>fsutil reparsepoint query Python34
    Reparse Tag Value : 0xa0000003
    Tag value: Microsoft
    Tag value: Name Surrogate
    Tag value: Mount Point
    Substitue Name offset: 0
    Substitue Name length: 58
    Print Name offset:     60
    Print Name Length:     50
    Substitute Name:       \??\C:\Program Files\Python34
    Print Name:            C:\Program Files\Python34

    Reparse Data Length: 0x00000078
    Reparse Data:
    0000:  00 00 3a 00 3c 00 32 00  5c 00 3f 00 3f 00 5c 00  ..:.<.2.\.?.?.\.
    0010:  43 00 3a 00 5c 00 50 00  72 00 6f 00 67 00 72 00  C.:.\.P.r.o.g.r.
    0020:  61 00 6d 00 20 00 46 00  69 00 6c 00 65 00 73 00  a.m. .F.i.l.e.s.
    0030:  5c 00 50 00 79 00 74 00  68 00 6f 00 6e 00 33 00  \.P.y.t.h.o.n.3.
    0040:  34 00 00 00 43 00 3a 00  5c 00 50 00 72 00 6f 00  4...C.:.\.P.r.o.
    0050:  67 00 72 00 61 00 6d 00  20 00 46 00 69 00 6c 00  g.r.a.m. .F.i.l.
    0060:  65 00 73 00 5c 00 50 00  79 00 74 00 68 00 6f 00  e.s.\.P.y.t.h.o.
    0070:  6e 00 33 00 34 00 00 00                           n.3.4...

fsutil reparsepoint:
http://technet.microsoft.com/en-us/library/cc785451.aspx
History
Date User Action Args
2014-05-07 00:33:11eryksunsetmessages: + msg218030
2014-05-06 11:24:20tim.goldensetmessages: + msg217976
2014-05-06 11:16:11eryksunsetmessages: + msg217975
2014-05-06 06:08:29terry.reedysetmessages: + msg217969
2014-05-06 04:02:06Kim.Gräsmansetmessages: + msg217966
2014-05-05 20:58:01tim.goldensetmessages: + msg217955
2014-05-05 20:20:26eryksunsetnosy: + eryksun
messages: + msg217953
2014-05-05 20:01:05tim.goldensetstatus: open -> closed
resolution: fixed
messages: + msg217949

stage: patch review -> resolved
2014-05-05 20:00:02python-devsetmessages: + msg217948
2014-05-05 19:19:38tim.goldensetmessages: + msg217943
2014-05-05 18:59:29zach.waresetnosy: + zach.ware
messages: + msg217941
2014-05-05 18:47:52python-devsetmessages: + msg217940
2014-04-29 18:58:47Kim.Gräsmansetfiles: + unlink_junctions_r2.patch

messages: + msg217538
2014-04-29 08:25:54Kim.Gräsmansetmessages: + msg217488
2014-04-29 07:48:00tim.goldensetmessages: + msg217479
2014-04-29 05:55:35Kim.Gräsmansetmessages: + msg217471
2014-04-29 04:26:20Kim.Gräsmansetmessages: + msg217464
2014-04-29 04:01:14Kim.Gräsmansetmessages: + msg217463
2014-04-28 19:28:56tim.goldensetmessages: + msg217404
2014-04-27 21:04:38Kim.Gräsmansetmessages: + msg217318
2014-04-27 19:27:19tim.goldensetmessages: + msg217312
2014-04-27 19:24:21Kim.Gräsmansetmessages: + msg217311
2014-04-27 17:38:31tim.goldensetmessages: + msg217293
2014-04-27 17:03:01python-devsetmessages: + msg217288
2014-04-27 17:00:29python-devsetnosy: + python-dev
messages: + msg217287
2014-04-16 20:16:44tim.goldensetmessages: + msg216579
2014-03-12 19:10:50Kim.Gräsmansetmessages: + msg213278
2014-02-11 19:58:56Kim.Gräsmansetmessages: + msg211008
2014-01-19 21:25:52Kim.Gräsmansetmessages: + msg208500
2014-01-02 10:47:21pitrousetstage: test needed -> patch review
versions: + Python 3.5, - Python 3.4
2014-01-02 10:45:41tim.goldensetmessages: + msg207157
2013-12-30 15:51:47Kim.Gräsmansetfiles: + unlink-junctions.patch

messages: + msg207097
2013-10-23 18:56:48tim.goldensetmessages: + msg201055
2013-10-23 18:55:15Kim.Gräsmansetmessages: + msg201054
2013-10-23 11:53:53hayposetnosy: - haypo
2013-10-23 11:42:33tim.goldensetmessages: + msg201018
2013-10-16 12:31:36Kim.Gräsmansetmessages: + msg200058
2013-09-29 20:59:55tim.goldensetassignee: tim.golden
messages: + msg198649
2013-09-29 20:34:28Kim.Gräsmansetfiles: + unlink_junction.patch
keywords: + patch
messages: + msg198647
2013-09-27 19:46:53Kim.Gräsmansetmessages: + msg198500
2013-09-25 20:52:38terry.reedysetversions: + Python 3.4, - Python 3.3
nosy: + terry.reedy

messages: + msg198416

type: behavior -> enhancement
stage: test needed
2013-09-25 20:20:30terry.reedysetnosy: + tim.golden, brian.curtin
2013-08-14 04:40:22Kim.Gräsmansetmessages: + msg195124
2013-08-14 03:48:40tim.peterssetnosy: + tim.peters
messages: + msg195121
2013-08-14 00:57:11hayposetnosy: + haypo
messages: + msg195114
2013-07-02 07:25:00Kim.Gräsmansetmessages: + msg192177
2013-06-27 12:21:52Kim.Gräsmansetmessages: + msg191946
2013-06-27 12:15:21Kim.Gräsmancreate