The Win32JunctionTests class fails when the test suite is run on an ImDisk[1]_ virtual disk.  The junctions are created successfully, however os.stat() fails on them (winerror 123).  os.lstat() does succeed.

I'm inclined to believe that this is a bug in the ImDisk device driver, but when testDown() is run, it fails to remove the newly created junction to the test directory.  By leaving the junction in place, when the test runner completes it removes the entire temporary test directory containing the junction thus removing the Lib test directory!

I suggest that at least changing the tearDown() method to use os.path.lexists() to ensure that the junction is removed regardless of its target existing or not.


.. [1] http://www.ltr-data.se/opencode.html/#ImDisk

Maybe the junction must not include Lib/ but only temporary directories? I
didn't read the tedt yet.

> I'm inclined to believe that this is a bug in the ImDisk device driver

Junctions, and other filesystem reparse points, are implemented by volume devices, not disk devices. NTFS and ReFS support reparse points, but FAT, FAT32, and exFAT do not. Does the current filesystem claim to support reparse points? Here's how to check this:

    import ctypes
    kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)

    FILE_SUPPORTS_REPARSE_POINTS = 0x00000080

    def volume_supports_reparse_points(root_path):
        flags = ctypes.c_uint()
        if not kernel32.GetVolumeInformationW(root_path,
                                              None, 0, None, None,
                                              ctypes.byref(flags),
                                              None , 0):
            raise ctypes.WinError(ctypes.get_last_error())
        return bool(flags.value & FILE_SUPPORTS_REPARSE_POINTS)

For example:

    >>> volume_supports_reparse_points('C:\\')
    True

Win32JunctionTests creates a junction to the "Lib/test" directory in the current directory. It should create a temporary target directory instead of using "Lib/test". 

I installed ImDisk Virtual Disk 2.0.9 and was able to create a valid junction on an NTFS RAM disk. What are the steps required to reproduce the problem?

To reproduce:

P:\python-default>PCBuild\amd64\python_d.exe
Python 3.6.0a0 (default:708beeb65026, Mar 30 2016, 08:50:27) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, _winapi
>>> _winapi.CreateJunction('PCbuild', 'junctest')
>>> assert os.listdir('PCbuild') == os.listdir('junctest')
>>> os.path.exists('junctest')
False
>>> os.path.lexists('junctest')
True
>>> os.stat('junctest')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [WinError 1] Incorrect function: 'junctest'
>>> os.lstat('junctest')
os.stat_result(st_mode=16895, st_ino=1688849860290629, st_dev=719101600, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1459349589, st_mtime=1459349589, st_ctime=1459349589)

I see now. It's not a problem with the junction. In the stat implementation, after verifying that the directory or file is a reparse point, it opens it again without FILE_FLAG_OPEN_REPARSE_POINT, in order to open the target instead of the link. I don't understand why it doesn't use this handle to call GetFileInformationByHandle again, this time on the target. Instead it calls get_target_path in order to recursively call stat on the target path. 

To get the target path it calls GetFinalPathNameByHandle and requests the DOS name, which requires a drive letter. At this point Windows has a native NT path (e.g. \Device\ImDisk0\python-default\PCBuild), so it has to query the mountpoint manager to get the DOS drive letter mapping (i.e. \Device\ImDisk0 => P:). This is done by opening a handle for \\.\MountPointManager and calling DeviceIoControl to query the drive letter. This is the call that fails with ERROR_INVALID_FUNCTION (1). Indeed, you'll get the same error if you try calling os.path._getfinalpathname on any directory or file on this drive.

It's probably the case that the ImDisk driver doesn't support the mountpoint manager at all. For example, in the \Global?? object directory there's no GUID symbolic link assigned to the device, which is normally assigned by the mountpoint manager. Also, mountvol.exe doesn't list the ImDisk drive.

I'm fine with the tests for CreateFunction failing for an ImDisk virtual drive, however something needs to be changed with the test to not remove the test directory on tearDown().

Changing it to use a temporary directory to link against is a workaround, but still leaves remnants once finished.  My solution of using lexists() in tearDown() however, ensures that any created items are cleaned up regardless of the test result.

I don't disagree with using os.path.lexists. However, I think it should also use a temporary target directory. Also, if it's possible to fix the behavior of os.stat when GetFinalPathNameByHandle fails (considering we already have a handle for the target), then that's icing on the cake. 

Another issue related to leaving garbage behind is when the DeviceIoControl call fails to store the reparse point. In this case, _winapi.CreateJunction fails to remove the directory that it creates for the junction. That should be fixed and tested. If CreateJunction fails with ERROR_INVALID_FUNCTION, test_create_junction should be modified to fail only if self.junction still exists. Otherwise the test should pass, i.e. it worked correctly by raising an exception and cleaning up after itself given a filesystem that doesn't support junctions.

os.stat() was redesigned in issue 37834, which entailed extensive updates across the standard library to improve support for Windows reparse points. As part of this,  Win32JunctionTests.tearDown() was changed to use a more reliable lexists() check, which resolves this issue.

FYI, the new implementation of os.stat() supports an ImDisk virtual disk (v2.0.9 from 2015-12). In the following example, "junctest" is a mountpoint (junction) in an NTFS filesystem. The filesystem is mounted on an ImDisk device, as seen its VOLUME_NAME_NT (2) path:

    >>> flags = win32file.FILE_FLAG_OPEN_REPARSE_POINT
    >>> flags |= win32file.FILE_FLAG_BACKUP_SEMANTICS
    >>> h = win32file.CreateFile('junctest', 0, 0, None, 3, flags, None)
    >>> win32file.GetFinalPathNameByHandle(h, 2)
    '\\Device\\ImDisk0\\junctest'

stat() traverses the mountpoint:

    >>> os.stat('junctest').st_reparse_tag == 0
    True

lstat() opens the mountpoint:

    >>> os.lstat('junctest').st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT
    True

This version of Imdisk doesn't support the mountpoint manager, so trying to get the VOLUME_NAME_DOS (0) name of r"\Device\ImDisk0" (e.g. r"\\?\R:") still fails the same as before:

    >>> win32file.GetFinalPathNameByHandle(h, 0)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    pywintypes.error: (1, 'GetFinalPathNameByHandle', 'Incorrect function.')

But os.stat() no longer needs it.