classification
Title: os.stat gives exception for Windows junctions in v3.3
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.3, Python 3.4
process
Status: closed Resolution: works for me
Dependencies: Superseder:
Assigned To: tim.golden Nosy List: John.Jefferies, christian.heimes, loewis, tim.golden, vstinner
Priority: normal Keywords: 3.3regression

Created on 2013-06-26 08:45 by John.Jefferies, last changed 2013-10-27 17:27 by tim.golden. This issue is now closed.

Messages (10)
msg191899 - (view) Author: John Jefferies (John.Jefferies) Date: 2013-06-26 08:44
If os.stat is executed on a Windows junction with Python 3.3 I see an exception:
------------
>>> import os
>>> os.stat('C:\Windows\System32\config\systemprofile\SendTo')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'C:\\Windows\\System32\\config\\systemprofile\\SendTo'
------------

whereas with Python 3.2 it works without error:
------------
>>> import os
>>> os.stat('C:\Windows\System32\config\systemprofile\SendTo')
nt.stat_result(st_mode=16895, st_ino=281474977136630, st_dev=0, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1295885671, st_mtime=1295885671, st_ctime=1295885671)
------------

FTR. Some background:
It's a pity that Python doesn't just treat Windows junctions as a normal soft link. But given that islink() returns False for a junction, I was able to work around that in Python 3.2 like this:
------------
if os.path.islink(fullname) or \
    os.stat(fullname)[stat.ST_INO] != os.lstat(fullname)[stat.ST_INO]:
    # If it's not a link, it's probably a junction...
------------

Many thanks for looking.

John
msg191903 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-26 11:29
Let's have a look
msg191904 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-26 12:38
On my Windows box (Win 7) I'm getting an error with Python 3.2 and 3.3. It looks like I'm not allowed to enter the parent directory (Zugriff verweigert == Permission Denied):

>>> os.stat(r'C:\Windows\System32\config')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
WindowsError: [Error 5] Zugriff verweigert: 'C:\\Windows\\System32\\config'

In order to reproduce your problem anyway I have created a directory, a junction point and a symlink (as admin):

> mkdir testdir
> mklink /j testjunktion testdir
> mklink /d testlinkk testdir

Neither os.stat() nor os.lstat() have failed with an exception. There must be something different going on on your system.


By the way junction points are soft links but not symbolic links. They are implemented as a different kind of NTFS reparsing points. Python should not treat a junction point as a link but have yet another check for it.

>>> os.lstat(r'c:\users\heimes\testdir')
nt.stat_result(st_mode=16895, st_ino=58265320179130300, st_dev=3366304641, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1372247959, st_mtime=1372247959, st_ctime=1372247959)

>>> os.lstat(r'c:\users\heimes\testjunction')
nt.stat_result(st_mode=16895, st_ino=4785074604141776, st_dev=3366304641, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1372247983, st_mtime=1372247983, st_ctime=1372247983)

>>> os.lstat(r'c:\users\heimes\testlink')
nt.stat_result(st_mode=41471, st_ino=12384898975270541, st_dev=3366304641, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1372249830, st_mtime=1372249830, st_ctime=1372249830)


PS: You should use raw strings on Windows or escape the backslashes.
msg191905 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2013-06-26 12:55
There is much confusing about junction point on the internet. Some sites like Wikipedia claim that a junction point is a kind of soft link. Other sites refer to junction points as "directory hard links".

IMO "directory hard links" is wrong. Contrary to file hard links a junction point doesn't keep the target directory alive when the target is removed.
msg191906 - (view) Author: John Jefferies (John.Jefferies) Date: 2013-06-26 14:13
On 26/06/2013 13:38, Christian Heimes wrote:
> Christian Heimes added the comment:
>
> On my Windows box (Win 7) I'm getting an error with Python 3.2 and 3.3. It looks like I'm not allowed to enter the parent directory (Zugriff verweigert == Permission Denied):

Ah. You need to be an administrator to look at the system directories? 
[I'm also on Win7].

> Neither os.stat() nor os.lstat() have failed with an exception. There 
> must be something different going on on your system. 

I didn't look closely enough; only the junctions in system folders are 
failing for me with an exception. The junctions in my user directory are 
not. I do hope I haven't wasted too much of your time in not describing 
the problem correctly.

> By the way junction points are soft links but not symbolic links. They 
> are implemented as a different kind of NTFS reparsing points. Python 
> should not treat a junction point as a link but have yet another check 
> for it. 

I realise there are differences between junctions and symlinks at the 
ntfs level, but the only things that seem relevant to the application 
are that junctions are limited to directories on a local volume. So I 
consider junctions to be a subset of symlinks. Regardless, any reliable 
indication that a directory is a junction would be useful, and the hack 
I've been using thus far appeared to work for all the junctions on my 
system.

> PS: You should use raw strings on Windows or escape the backslashes.

Yes, I'm ashamed, I did know that but missed it from the example.

Many thanks.

John
msg194046 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-08-01 11:09
I propose to close this one: using Python 3.3 on Win7 I can successfully stat NTFS Junctions. Is there any remaining issue?
msg194116 - (view) Author: John Jefferies (John.Jefferies) Date: 2013-08-01 20:41
On 01/08/2013 12:09, Tim Golden wrote:
> Tim Golden added the comment:
>
> I propose to close this one: using Python 3.3 on Win7 I can successfully stat NTFS Junctions. Is there any remaining issue?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue18306>
> _______________________________________

The original issue remains for me on v3.3.2. I can stat junctions in my 
user directory, but not in system directories. In v3.2 I can do that 
provided I am running with elevated privilege. If it's fixed in the next 
version that's great.

------------
Python 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 
bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
 >>>
 >>> import os
 >>> os.stat(r'C:\Windows\System32\config\systemprofile\SendTo')
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
FileNotFoundError: [WinError 2] The system cannot find the file 
specified: 'C:\\Windows\\System32\\config\\systemprofile\\SendTo'
 >>>
------------

Regards

John
msg201098 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-10-24 08:35
Just revisited this to see if I could close off. One thing occurred to me which should have come up before: this situation will be aggravated by WOW64 file redirection. 

If I run 64-bit Python on 64-bit Windows I can successfully stat links in %windir%\system32 under 3.4 and 3.3. If I run 32-bit Python on 64-bit Windows, I get the error you describe because Windows will silently redirect the 32-bit Python process towards %windir%\SysWow64 where the junction (or, possibly, its target) does not exist.

Given that I'm able to stat %windir%\system32 junctions on both versions, I'm going to close this as works-for-me. I think there's a case for a bigger-picture look at the resolution of reparse points across the stdlib but that would have to be its own issue.
msg201437 - (view) Author: John Jefferies (John.Jefferies) Date: 2013-10-27 08:58
Hello Tim,

The fact that it works on 64-bit Python obviously reduces the priority. 
I will make a point of choosing the 64-bit version in the future.

FWIW. I'm dubious about the problem being solely attributable to WOWs 
handling of junctions because my Python 3.2 (that doesn't have the 
problem) is also 32-bit. There must be something else, even if that 
something isn't going to be fixed.

Thankyou for spending time on this.

John

On 24/10/2013 09:35, Tim Golden wrote:
> Tim Golden added the comment:
>
> Just revisited this to see if I could close off. One thing occurred to me which should have come up before: this situation will be aggravated by WOW64 file redirection.
>
> If I run 64-bit Python on 64-bit Windows I can successfully stat links in %windir%\system32 under 3.4 and 3.3. If I run 32-bit Python on 64-bit Windows, I get the error you describe because Windows will silently redirect the 32-bit Python process towards %windir%\SysWow64 where the junction (or, possibly, its target) does not exist.
>
> Given that I'm able to stat %windir%\system32 junctions on both versions, I'm going to close this as works-for-me. I think there's a case for a bigger-picture look at the resolution of reparse points across the stdlib but that would have to be its own issue.
>
> ----------
> assignee:  -> tim.golden
> resolution:  -> works for me
> stage: test needed -> committed/rejected
> status: open -> closed
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue18306>
> _______________________________________
msg201478 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2013-10-27 17:27
If you can put together a self-contained test case which fails on 
3.3/3.4 I'm quite happy to re-open this. Given the nature of the issue, 
you'll have to be quite precise as to Windows version, whether 32/64, 
whether running as Administrator, and exact Python version.

Feel free to call mklink or junction to create the link so that it's 
very clear what's linking to what and where.
History
Date User Action Args
2013-10-27 17:27:10tim.goldensetmessages: + msg201478
2013-10-27 08:58:45John.Jefferiessetmessages: + msg201437
2013-10-24 08:35:40tim.goldensetstatus: open -> closed
messages: + msg201098

assignee: tim.golden
resolution: works for me
stage: test needed -> resolved
2013-08-01 20:41:45John.Jefferiessetmessages: + msg194116
2013-08-01 11:09:35tim.goldensetmessages: + msg194046
2013-06-26 21:07:46vstinnersetnosy: + vstinner, tim.golden
components: + Windows
2013-06-26 14:13:45John.Jefferiessetmessages: + msg191906
2013-06-26 12:55:11christian.heimessetmessages: + msg191905
2013-06-26 12:38:29christian.heimessetassignee: christian.heimes -> (no value)

messages: + msg191904
nosy: + loewis
2013-06-26 11:29:37christian.heimessetassignee: christian.heimes
versions: + Python 3.4
keywords: + 3.3regression
nosy: + christian.heimes

messages: + msg191903
stage: test needed
2013-06-26 08:45:00John.Jefferiescreate