Python reports an import error, even when a valid package exists on
sys.path, if that package is a symlink directory and would otherwise be
importable.

The attached test script demonstrates the limitation on Python 2.6.2 and
3.1.0.

I suspect that the underlying import routines have some assumptions
about the file system that exclude symlinked directories.

Given that you had to use ctypes to create the symlinks on Windows in
the first place, I'd say you're right :)

Probably the first step would be to make sure the nt OS and path modules
get Windows symlink support (although even that may not be enough, due
to the bootstrapping issues with file access and the import machinery).

I agree that a proper solution might include full symlink support.

However, symlink support for windows (issue1578269) is taking some time
to develop and will likely never be available for Python 2.

My feeling is that a symlink naive implementation (such as Python is
now), should treat symlinks in a naive manner -- that is, treat them as
regular files and directories and let the OS perform the abstraction.

I hope to begin investigating this after I've closed out issue1578269,
although I'd be happy if someone else wanted to address this issue
beforehand.

If it's possible to address this in a straightforward way independent of
a symlink-aware Python interpreter, that would be desirable.

Symlinked packages used to work on Windows until recently, but a few days ago Microsoft published a few security patches and things stopped working. On my local machine I uninstalled all Visual C++ Runtime library patches (they were installed automatically on May 5th) and symlinks started working on my machine, again. On my friend's computer symlinked packages suddenly stopped working today, but he doesn't seem to have the Visual C++ Runtime libs, so we don't know how to work around this issue on his computer. Could someone please fix support for symlinked packages? It's an essential feature during development.

Are you able to narrow it down to which security update(s) caused the breakage?

I uninstalled these three security patches:

* KB2467173: http://support.microsoft.com/kb/2467173
* KB2467174: http://support.microsoft.com/kb/2467174
* KB2467175: http://support.microsoft.com/kb/2467175

Without those patches symlinked packages work for me. I didn't try to narrow it down to the exact patch. Instead I uninstalled all Visual C++ Redistributable packages in order to prevent those patches from ever being installed again. Should I risk breaking symlink support again to find the source of the problem?

Don't do anything you're not comfortable with. If you can get your system to whatever state it was in the past where things worked properly, feel free to dig into it. I will try to look into this situation and see if there's anything in these security updates that looks related.

That's interesting. You're saying that you've been using symlinked packages for some time and that it works for you. I filed this bug because it's never worked for me. Can you describe a little bit about the environment in which you've encountered this? Specifically, which Python version, how were the symlinks created, and what is the sys.path like?

And since you seem to have some systems that honor symlinked packages, can you run the attached test_import_symlink_package.py and report the results?

Out of the patches listed, http://www.microsoft.com/technet/security/advisory/2269637.mspx is the only part that appears to be related in any way, although it doesn't specify a whole lot on the surface. The title is "Insecure Library Loading Could Allow Remote Code Execution", so I'm guessing they could have made some changes under the hood that could affect us.

Before we go down that road, running the script Jason suggested would be helpful in proving what works and doesn't work depending on what patches are installed and what version is running.

I'm using Python 2.6.6 on Windows 7 Professional with the latest service pack. My system is pretty bare bones. Do you have Visual Studio or the Visual Studio Redistributables?

Here's the output of your script (the last exception seems to be caused by the .pyc file, but apart from that it seems to work):


C:\Users\wkornewald\Downloads\a>test_import_symlink_package.py
Problem does not exist here.
Traceback (most recent call last):
  File "C:\Users\wkornewald\Downloads\a\test_import_symlink_package.py", line 66, in <module>
    os.rmdir(tagged)
WindowsError: [Error 145] The directory is not empty: 'sample-tagged'

C:\Users\wkornewald\Downloads\a>dir sample-tagged
 Volume in drive C has no label.
 Volume Serial Number is 50C8-BE90

 Directory of C:\Users\wkornewald\Downloads\a\sample-tagged

10.05.2011  19:04    <DIR>          .
10.05.2011  19:04    <DIR>          ..
10.05.2011  19:04               107 __init__.pyc
               1 File(s)            107 bytes
               2 Dir(s)  66.517.442.560 bytes free

Thanks for that. The output is very telling.

First, it shows there's a bug in the test script where an exception occurs when it succeeds because a .pyc file is created and not properly cleaned up.

Second, it demonstrates that the bug as I reported it is not occurring in your environment.

This latter result tells me that I may be experiencing the same issue as you, but encountered it almost two years ago. This finding leads me to believe that the issue is less about the latest hotfix and more related to the VS redistributables. I almost certainly have some form of these redistributables installed as I usually have some form of Visual Studio installed.

The fact that it works in one environment and not another means I should be able to find the regression. I'll install a barebones Windows box and see how it behaves there.

I updated the test script to avoid the error when the .pyc (or __pycache__) is created.

Confirmed the issue exists on Python 2.7 and 3.2

Updated script to run under Python 3.2 as well.

I've tracked it down. It's caused by KB2467174 (http://support.microsoft.com/kb/2467174) and I can reliably reproduce it. Here's how to reproduce it:

Install *only* the "Microsoft Visual C++ 2008 Redistributable Package (x86)" from http://www.microsoft.com/downloads/en/confirmation.aspx?familyid=9b2da534-3e03-4391-8a4d-074b9f2bc1bf&displaylang=en

Then, start Windows Update and click "Check for Updates" to get the new C++ 2008 Redistributable update and install that patch. Immediately symbolic links stop working. Then uninstall that patch and symlinks work again.

We're still trying to figure out what's causing this problem on my friend's computer. He doesn't have the 2008 Redistributable.

That's really strange. We uninstalled Service Pack 1 from my friend's machine and now everything works. What I don't understand is why I don't have Service Pack 1 on my machine and why Windows Update also doesn't offer to install it. Does it have to be installed manually? :-/

Anyway, this means that the problem can be caused either by SP1 or by KB2467174.

I've made some progress on this issue. Thanks to Waldemar's findings, I'm able to selectively reproduce the issue by installing/uninstalling the Visual C++ redistributable that includes KB2467174 on a Windows 7 RTM installation.

So I added a patch to cpython to help troubleshoot:

    PS C:\Users\jaraco\projects\public\cpython\pcbuild> hg id
    434dfe42fde1+
    PS C:\Users\jaraco\projects\public\cpython\pcbuild> hg diff
    diff -r 434dfe42fde1 Python/import.c
    --- a/Python/import.c   Sun May 15 13:16:22 2011 +0200
    +++ b/Python/import.c   Mon May 16 10:52:45 2011 -0400
    @@ -1807,6 +1807,15 @@
         /* Check for package import (buf holds a directory name,
            and there's an __init__ module in that directory */
     #ifdef HAVE_STAT
    +    if (Py_VerboseFlag > 1 && _Py_stat(filename, &statbuf) == 0) {
    +        PySys_FormatStderr("stat_mode %U: %d\n",
    +                           filename, statbuf.st_mode);
    +    }
    +    else if(Py_VerboseFlag > 1)
    +    {
    +        PySys_FormatStderr("_Py_stat(%U) returned %d\n",
    +            filename, _Py_stat(filename, &statbuf));
    +    }
         if (_Py_stat(filename, &statbuf) == 0 &&         /* it exists */
             S_ISDIR(statbuf.st_mode))           /* it's a directory */
         {


I then ran the test script without vcredist:

    C:\Users\Jason R. Coombs\Desktop>u:\users\jaraco\projects\public\cpython\pcbuild\amd64\python -vv .\test_import_symlink_package.py   2>&1  | findstr sample
    stat_mode .\sample: 16895
    import sample # directory '.\\sample'
    _Py_stat(.\sample\__init__) returned -1
    # trying '.\\sample\\__init__.pyd'
    # trying '.\\sample\\__init__.py'
    import sample # from '.\\sample\\__init__.py'
    # wrote '.\\sample\\__pycache__\\__init__.cpython-32.pyc'
    # cleanup[1] sample

And again with the vcredist installed:

    C:\Users\Jason R. Coombs\Desktop>u:\users\jaraco\projects\public\cpython\pcbuild\amd64\python -vv .\test_import_symlink_package.py   2>&1  | findstr sample
    _Py_stat(.\sample) returned -1
    # trying '.\\sample.pyd'
    # trying '.\\sample.py'
    # trying '.\\sample.pyw'
    # trying '.\\sample.pyc'
    No module named 'sample'

Basically, it appears that with the later vcredist installed, _Py_stat is returning -1 for symlink directories.

I decided to investigate further. I created another script to test the call to _wstat to try to recreate the -1 return code, but I was unable to do so. I'm attaching the script, which creates the same 'sample' package, but instead of calling import, calls _wstat directly on the 'sample', but it returns 0, even in the environments where the test script fails.

This leads me to believe the issue isn't with _wstat, but with the way _wstat is called in _Py_stat.

Digging deeper with the Visual Studio debugger, I discovered the following interesting outcome (run with cmd.exe):

    @echo off
    mklink /d sample sample-target
    mkdir sample-target
    echo "" > sample-target\__init__.py
    :: Before KB2467174, returns 0; after, returns -1
    python -c "import ctypes; buf = (ctypes.c_char*256)(); print(ctypes.windll.msvcr90._wstat64i32(u'sample', ctypes.byref(buf)))"
    :: Always returns 13
    python -c "import ctypes; fd = ctypes.c_int(-1); print(ctypes.windll.msvcrt._wsopen_s(ctypes.byref(fd), u'sample', 0, 0x40, 0))"
    rmdir sample
    rmdir /s /q sample-target

The call to _wstat64i32 is returning -1 with the patched CRT... but when I traced _wstat64i32 (on a machine with the patched CRT), it called _wsopen_s, which apparently always returns -1.

So my guess is that the patched CRT changed something about _wstat64i32 such that it doesn't defer to _wsopen_s. I'm going to see if I can get an early version of Visual Studio that I can trace where _wstat64i32 doesn't fail.

Indeed, this appears to be a bug in stat64.c, specifically a regression in KB2467174. If I look at the code for _wstat64i32, it doesn't have the code that calls into _wsopen_s for symlinked files/dirs, so uses the legacy behavior to stat the target. I suspect it's also a memory leak, because it doesn't properly close the "findhandle" in this case.

Any suggestions on how to handle upstream bugs in Windows?

Wow, nice analysis. http://connect.microsoft.com/ is the external Microsoft bug tracker, as far as I know.

MS Connect is apparently only for projects under active development, not mature, released products. I've posted to the MSDN forums, where with my MSDN account, I can expect priority support from MS personnel.

http://social.msdn.microsoft.com/Forums/en-US/vcgeneral/thread/93ebb061-d952-4650-b15c-30548a6649a8

I hope this helps resolve the issue at its source.

Per the suggestion in the Visual Studio forums, I posted a report against Visual Studio here: https://connect.microsoft.com/VisualStudio/feedback/details/669601/regression-calling-wstat64i32-on-symlink-directory-with-kb2467174-installed#details

Removing 2.6, this is not a security bug.

(OT:
> Could someone please fix support for symlinked packages? It's an
> essential feature during development.
If I correctly guess your use case, you could use a pth file during development.  See the docs.)

BTW, does this bug occur only with symlinked packages or also with one-file modules?

It was only with symlinked packages/folders. Symlinked files worked correctly for me.

BTW, pth files won't work for our specific use-case (App Engine).

If this issue is important to you, please follow the link above to the Microsoft Connect site and vote for the issue. More votes should mean it will get more attention. Thanks.

Is this bug going to be fixed? I've been experiencing this for some time now, and it seems the Python team has known about it for a while too... seems like an easy fix!

Not fixed, but if it's easy, you're welcome to fix it before we get around to it.

Well, I am making an assumption here. :-)

Might have a look at a patch. Never looked at the Python runtime before. If it's all written in C, then it's possible. I'm guessing most of the Python programmers don't know much/anything about Win32 programming, so that probably explains the delay.

There are a few of us, and Jason and myself have done most of the Windows symlink related work. We'll certainly get to this and have it fixed, but with no releases on the immediate horizon, there isn't a rush. This and your other symlink issue are on my radar for things to make sure we cover.

Thanks Brian; that's good to know. If we could get it in the next release (Python 2.8?) that would be awesome.

> If we could get it in the next release (Python 2.8?) that would be awesome.
I doubt it will (see PEP 404) but 3.3 is a good target.

I've filed a support incident with the Microsoft Support team to try to push this issue determine a fix or workaround.

That's great, thanks.

The MS support engineer has come back confirming the bug and suggesting using the MS C API instead of wstat for doing directory detection. It's still possible they'll fix the bug in an upcoming update, but even if they do, it will leave a window of broken systems, so I'm inclined to implement a workaround (i.e. not use stat for directory detection in import.c).

I started looking at implementing this for the Python 2.7 branch and I found that I could drastically simplify the code by extracting an 'isdir' function.

Consider the two changesets in this patch. I've confirmed this fixes the issue. Would somebody review this patch? If it's deemed acceptable, I'll consider something similar for Python 3.x.

Please regenerate your diff file, it undoes an unrelated fix :-)

That's very good news. I suspected MS has written a wrapper over this somewhere to accommodate for this bug, and it was just a matter of using the right API.

I look forward to seeing this in the next 2.7 release.

I've regenerated the diff a couple of times (once after rebasing to not undo Amaury's changes and another after correcting the whitespace).

I've created yet another patch (1cdb64480494) which adds a regression test under Python 2.7 to demonstrate that the fix works (based on test_import_symlink_package.py).

4aaf78f0dd10.diff is the same patch as that for 2.7 but ported to 3.1.

I ported the patch to the 3.2 branch. This port required more extensive adjustments. In particular, the NullImporter no longer used (char *) for directory detection, but instead used platform-specific PyObjects.

Therefore, the 3.2 diff creates a new function, Py_PathIsDir, which takes a PyUnicode on Windows and PyBytes on other platforms and calculates the directory state of the parameter.

I'm fairly unhappy with the way the patch for 2.7 and 3.1 translated to 3.2.

So, I'm seeking review and approval of the 2.7 and 3.1 patches, and comments on the 3.2 patch.

(3.1 doesn’t get non-security bug fixes either)

After hearing back from Microsoft support (and by proxy, the Visual Studio development team), it is clear that this issue is very low priority for them (they see it as having trivial business impact), so we cannot expect it to be fixed in the upstream libraries anytime soon.

This response essentially means that if Python wants to support these symbolically-linked directories, it cannot use stat/wstat on Windows and must use instead the Windows APIs.

As seen in issue13412, this bug not only affects import.c, but also affects posixmodule (such as with os.listdir). After addressing this issue for import.c, it will probably be necessary to also address it for other parts of Python that use stat/wstat (or other calls that depend on those calls).

New changeset 6c218b9c5c4c by Jason R. Coombs in branch '2.7':
Extracted Windows directory detection from NullImporter.__init__. This greatly simplifies the code and fixes issue6727.
http://hg.python.org/cpython/rev/6c218b9c5c4c

New changeset 92f4d4eebed1 by Jason R. Coombs in branch '2.7':
Adding regression test for issue6727
http://hg.python.org/cpython/rev/92f4d4eebed1

I've gone ahead and pushed the changesets for Python 2.7. I'll continue to hold off on Python 3.2/3.3 until I can review. Perhaps Brian and I can review these changes at PyCon.

It breaks the compilation on few buildbots.


  Python/import.c:130:14: error: #if with no expression

I see that. Thanks. I'm on it.

The Unix tests are now passing again. Some Windows buildbots are failing because they do not have the symlink privilege. I'll add a guard in the test to not run it if it doesn't have privilege to symlink.

The @support.skip_unless_symlink decorator could be helpful there.

At PyCon, I took another look at the Python 3.2 changes. Unfortunately, I found they trigger failures in other aspects of the test suite, so the approach is not viable as implemented.

I plan to take another look at this at some point, but I've been utterly swamped. If someone wants to take a look at the 3.2 branch of my bitbucket clone, I would have no problem with that. Otherwise, I'll try to take a look at it when I can.

New changeset afe67ea94bc6 by Jason R. Coombs in branch 'default':
Adding test from issue6727 demonstrating that symlink import issue does not occur here in 3.3
http://hg.python.org/cpython/rev/afe67ea94bc6

I don't know what changed in 3.3 to address this issue; likely candidates are the VS 2010 upgrade or the rewrite of the import machinery. In either case, best I can tell, 3.3 is no longer implicated.

It seems that by adding the test to the default branch, the buildbots are now broken for Unix systems (they're failing the test). I'm going to await the results from the Windows buildbots, and then skip the tests on the failing platforms.

I've disabled the test on non-Windows systems and created issue15091 to track the issue.

Would you mind making a small change and use assertEqual instead of an assert statement?  The test suite should run the same tests under -O and -OO; there are remaining asserts in old tests but for the last year or two we’ve tried to change them.  Thanks in advance.

(Oh BTW I put back 3.3 in versions without seeing you had removed it recently; even though the bug is not present there, the tests from 3.2 are merged into 3.3 so if I understand the use of the versions field correctly it makes sense to keep 3.3.  Didn’t want you to think I was reverting your deliberate changes on a whim :)

New changeset 24369f6c4a22 by Jason R. Coombs in branch 'default':
Prefer assertEqual to simply assert per recommendation in issue6727.
http://hg.python.org/cpython/rev/24369f6c4a22

As far as I can tell from the messages, there is nothing left to do here, so I'm closing it.