This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Symbolic links omitted by os.listdir on some systems
Type: behavior Stage:
Components: IO, Windows Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: alexreg, arigo, brian.curtin, eric.smith, jaraco, tim.golden
Priority: normal Keywords:

Created on 2011-11-15 22:36 by alexreg, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (16)
msg147715 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 22:36
Python 2.7 has no knowledge of directory symlinks on Windows 7. Listing a directory symlink does not work, nor does accessing a file within one. This is quite a notable missing feature on Windows 7, where symlinks are becoming increasingly prevalent.
msg147721 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-11-15 22:54
Python 2.7 is not aware of symlinks and treats them like their targets, so it should be able to list a symlink directory or access a file within one. For example:

PS C:\Users\jaraco> cmd /c dir
 Volume in drive C is system
 Volume Serial Number is 2455-92A0

 Directory of C:\Users\jaraco

11-Nov-2011  06:50    <DIR>          .
11-Nov-2011  06:50    <DIR>          ..
..
11-Aug-2011  10:11    <SYMLINKD>     bin [Dropbox\bin\x64]
10-Aug-2011  21:00    <SYMLINKD>     bin-x86 [Dropbox\bin\x86]
..
              11 File(s)          6,632 bytes
              27 Dir(s)  301,972,422,656 bytes free
PS C:\Users\jaraco> python
Python 2.7.2 (default, Jun 12 2011, 14:24:46) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir('bin')
['curl.exe', 'devcon.exe', 'HotSwap!.EXE', 'Notepad2.exe', 'utorrent.exe', 'vncviewer.exe']
>>> with open('bin/curl.exe', 'rb') as f:
...   f.read()[:10]
...
'MZ\x90\x00\x03\x00\x00\x00\x04\x00'
>>> ^Z

Because Python 2.7 is feature-frozen, symlink support will not be available in Python before 3.2.

If you are experiencing errors, please reopen this ticket and describe what you are doing, what you expect, and what you get instead.
msg147723 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 23:02
What are you running? This is not what I get on Win7 x64, and I have had several other users in ##python on FreeNode confirm this inability. As far as Python is concerned, these dir sym links do not even exist.
msg147724 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-11-15 23:02
I think we could still make os.listdir work properly. I'll look into a patch for this.

One "problem" here is the testability, since we'd need to rely on the mklink CLI app to create the symlinks, which requires that the calling application (python.exe) has elevated privileges. It wont be exercised by any of the buildbots.
msg147725 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-11-15 23:06
I should mention that there is third-party symlink support in jaraco.windows (http://pypi.python.org/pypi/jaraco.windows). Just easy_install it, and then use jaraco.windows.filesystem.symlink.

If there are features you need for symlink support in jaraco.windows, request them via https://bitbucket.org/jaraco/jaraco.windows and they should be implemented quickly.
msg147726 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 23:07
Thanks Bryan, that would be great.

The elevated privs problem could potentially be avoided by creating symlinks using the Win32 API directly. As long as the appropiate group policy is set, one does not require admin privs to create symlinks. Perhaps symlink creation/deletion on Windows could even be integrated into the Python class library?
msg147727 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-11-15 23:07
Brian, I'm still not sure I see the problem with os.listdir. It includes symlinks when listing a dir and traverses them naturally when referencing them as part of a path to listdir. Under what conditions does it fail?
msg147728 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 23:10
Well, at the very least it's a system-dependent issue, since I've tried out listdir and also file access on my system and some other Windows users'.
msg147729 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-11-15 23:18
Some variables could be a missing privilege or role, or perhaps a UAC restriction. What error do you get when you attempt to invoke os.listdir on a symlink directory?
msg147730 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 23:19
No error whatsoever. Python just thinks it doesn't exist unfortunately. Same report from other users...
msg147733 - (view) Author: Brian Curtin (brian.curtin) * (Python committer) Date: 2011-11-15 23:41
symlinks when listing a dir and traverses them naturally when referencing
them as part of a path to listdir. Under what conditions does it fail?
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue13412>
> _______________________________________

Jason - I'm responding on a phone - I haven't confirmed anything yet (and
hadn't yet seen your examples when I was typing).

Alex - the Windows API also requires elevation - that's what we have to do
in 3.2+ as well. The symlink calls require a specific privilege which is
only granted when elevated, even for the mklink program.
msg147734 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-15 23:42
When listing the parent dir in which a dir symlink resides, the dir symlink doesn't show up. That's the one I noticed most.
msg147736 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2011-11-15 23:48
On Windows Vista and Windows 7, there should be a symlink directory:

C:\Documents and Settings -> .\Users

Perhaps a good test would be:

assert 'Documents and Settings' in os.listdir('C:\\')

As long as there aren't any i18n issues, and the system is a standard install, that test should only fail if os.listdir fails to return symlinks.
msg147744 - (view) Author: Alex Regueiro (alexreg) Date: 2011-11-16 01:16
Okay, so I figured out where I was originally seeing this issue. (It does indeed work normally as you said; I'm not sure how I tested otherwise.)

Well, the problem is when you load a module that is a SYMLINKD. If the module folder is a normal directory, all works fine, otherwise it can't be found. Hopefully you can verify this.
msg147978 - (view) Author: Armin Rigo (arigo) * (Python committer) Date: 2011-11-20 09:24
I think this can then be considered a duplicate of issue #6727.
msg151383 - (view) Author: Jason R. Coombs (jaraco) * (Python committer) Date: 2012-01-16 16:31
While this may in fact be related to #6727, I don't believe it's a proper duplicate. I believe there's a more fundamental issue that's causing both #6727 and #13412, and that's that stat/wstat is broken on Windows due to a regression in the MS C runtime: http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/93ebb061-d952-4650-b15c-30548a6649a8/

I mention that here because a fix for #6727 if localized to import.c may not address this issue.
History
Date User Action Args
2022-04-11 14:57:23adminsetgithub: 57621
2012-01-16 16:31:31jaracosetmessages: + msg151383
2011-11-20 09:24:08arigosetstatus: closed

nosy: + arigo
messages: + msg147978

resolution: wont fix -> duplicate
2011-11-16 01:16:24alexregsetmessages: + msg147744
2011-11-15 23:49:44jaracosetstatus: closed -> (no value)
title: No knowledge of symlinks on Windows -> Symbolic links omitted by os.listdir on some systems
2011-11-15 23:48:40jaracosetmessages: + msg147736
2011-11-15 23:42:06alexregsetmessages: + msg147734
2011-11-15 23:41:15brian.curtinsetmessages: + msg147733
2011-11-15 23:19:23alexregsetmessages: + msg147730
2011-11-15 23:18:47jaracosetmessages: + msg147729
2011-11-15 23:10:09alexregsetmessages: + msg147728
2011-11-15 23:07:43jaracosetmessages: + msg147727
2011-11-15 23:07:05alexregsetmessages: + msg147726
2011-11-15 23:06:26jaracosetmessages: + msg147725
2011-11-15 23:02:50brian.curtinsetmessages: + msg147724
2011-11-15 23:02:07alexregsetmessages: + msg147723
2011-11-15 22:54:23jaracosetstatus: open -> closed
resolution: wont fix
messages: + msg147721
2011-11-15 22:44:49floxsettype: behavior
components: + Windows
2011-11-15 22:38:25eric.smithsetnosy: + jaraco, eric.smith
2011-11-15 22:37:10pitrousetnosy: + tim.golden, brian.curtin
2011-11-15 22:36:42alexregcreate