This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author eryksun
Recipients brett.cannon, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Date 2016-08-12.12:17:44
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1471004265.65.0.476387128688.issue27730@psf.upfronthosting.co.za>
In-reply-to
Content
I overlooked some aspects of the problem:

* A short relative path may end up exceeding MAX_PATH when normalized as a fully qualified path.
* The working directory may be a UNC path or may already have the \\?\ prefix. It's not thread-safe to check this beforehand.
* A path that contains a reserved DOS device name results in a \\.\ path.
 
Thus on second thought I think it's safer to call GetFullPathNameW for all paths that lack the \\?\ prefix, and then copy the result if it needs to be prefixed by \\?\ or \\?\UNC. The final path, if it's a filesystem path, should always use the \\?\ namespace to ensure that functions such as shutil.rmtree won't fail. 

For example:

    _DOS_DEVICES = "\\\\.\\"
    _NT_DOS_DEVICES = "\\\\?\\"
    _NT_UNC_DEVICE = "\\\\?\\UNC"

    def winapi_path(path):
        path = os.fsdecode(path) or '.'
        if path.startswith(_NT_DOS_DEVICES):
            return path
        temp = os.path._getfullpathname(path)
        if temp.startswith((_NT_DOS_DEVICES, _DOS_DEVICES)):
            return path if temp == path else temp
        if temp.startswith('\\\\'):
            return _NT_UNC_DEVICE + temp[1:]
        return _NT_DOS_DEVICES + temp

For reference, here's a typical call pattern when Windows 10.0.10586 converts a DOS path to an NT path:

    RtlInitUnicodeStringEx
    RtlDosPathNameToRelativeNtPathName_U_WithStatus
        RtlInitUnicodeStringEx
        RtlDosPathNameToRelativeNtPathName
            RtlGetFullPathName_Ustr
            RtlDetermineDosPathNameType_Ustr
            RtlAllocateHeap
            memcpy

RtlGetFullPathName_Ustr is called with a buffer that's sizeof(WCHAR) * MAX_PATH bytes. GetFullPathNameW also calls RtlGetFullPathName_Ustr, but with a caller-supplied buffer that can be up to sizeof(WCHAR) * 32768 bytes.

Here's the call pattern for a \\?\ path:

    RtlInitUnicodeStringEx
    RtlDosPathNameToRelativeNtPathName_U_WithStatus
        RtlInitUnicodeStringEx
        RtlDosPathNameToRelativeNtPathName
            RtlpWin32NtNameToNtPathName
                RtlAllocateHeap
                RtlAppendUnicodeStringToString
                RtlAppendUnicodeStringToString

RtlpWin32NtNameToNtPathName copies the path, replacing \\? with the object manager's \?? virtual DOS devices directory. 

Here's some background information for those who don't already know the basics of how Windows implements DOS devices in NT's object namespace, which you can explore using Microsoft's free WinObj tool.

In Windows NT 3 and 4 (before Terminal Services) there was a single \DosDevices directory, which is where the system created DOS device links to the actual NT devices in \Device, such as C: => \Device\HarddiskVolume2. Windows 2000 changed this in ways that were problematic, mostly due to using a per-session directory instead of a per-logon directory. (Tokens for multiple logon sessions can be used in a single Windows session, and almost always are since UAC split tokens arrived in Vista.) 

The design was changed again in Windows XP. \DosDevices is now just a link to the virtual \?? directory. The system creates DOS devices in a local (per logon) directory, except for system threads and LocalSystem logons (typically services), which use the \GLOBAL?? directory. The per-logon directories are located at \Sessions\0\DosDevices\[LogonAuthenticationId]. The object manager parses \?? by first checking the local DOS devices and then the global DOS devices. Each local DOS devices directory also has a Global link back to \GLOBAL??. It's accessible as \\?\Global\[Device Name], which is useful when a local device has the same name as a global one. The root directory of the object namespace is accessible to administrators using the \GLOBAL??\GLOBALROOT link, which from the Windows API is \\?\GLOBALROOT.
History
Date User Action Args
2016-08-12 12:17:45eryksunsetrecipients: + eryksun, brett.cannon, paul.moore, tim.golden, zach.ware, steve.dower
2016-08-12 12:17:45eryksunsetmessageid: <1471004265.65.0.476387128688.issue27730@psf.upfronthosting.co.za>
2016-08-12 12:17:45eryksunlinkissue27730 messages
2016-08-12 12:17:44eryksuncreate