classification
Title: os.listdir(...) on deep path on windows in python2.7 fails with errno 123
Type: behavior Stage: resolved
Components: Library (Lib), Windows Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Anthony Sottile, eryksun, paul.moore, serhiy.storchaka, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2018-01-12 16:56 by Anthony Sottile, last changed 2018-01-15 21:40 by serhiy.storchaka. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 5169 merged Anthony Sottile, 2018-01-12 17:27
Messages (4)
msg309862 - (view) Author: Anthony Sottile (Anthony Sottile) * Date: 2018-01-12 16:56
On windows, a deep path can be accessed by prefixing the path with \\?\

https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx?f=255&MSPPError=-2147217396#maxpath

The call to `listdir()` fails because it uses a posix separator.

A quick patch to fix this is to use `SEP` here: https://github.com/python/cpython/blob/ab95b3074ee43098edf3f23b07fb18ef57ee614d/Modules/posixmodule.c#L2388 (I can submit a patch for this)

Here's a stacktrace

>>> os.listdir(r'\\?\C:\Temp')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
WindowsError: [Error 123] The filename, directory name, or volume label syntax is incorrect: '\\\\?\\C:\\Temp/*.*'


This works fine in python3, as the code has been refactored to use `SEP` already.

>>> os.listdir(r'\\?\C:\Temp')
[]
msg309866 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2018-01-12 17:34
This should be fixed. That said, we have to use a unicode string for a long path anyway. Prior to Windows 8, the conversion from ANSI to Unicode in the system runtime library uses a static MAX_PATH buffer, so the ANSI API is inherently limited to MAX_PATH. You'll see this in the documentation of all functions that allow using \\?\ extended paths:

    In the ANSI version of this function, the name is limited to 
    MAX_PATH characters. To extend this limit to 32,767 wide 
    characters, call the Unicode version of the function and 
    prepend "\\?\" to the path.

Decoding ANSI strings uses a dynamically-sized buffer in Windows 8, but the change is undocumented and should not be relied upon.

Unfortunately, we can't reliably use a raw unicode string literal in Python 2, since \u and \U escapes are still evaluated. We can instead use forward slashes and normalize via os.path.normpath.
msg310017 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-15 21:39
New changeset 27f32e938ff51fd5d90a29abbbabc6b81d156f33 by Serhiy Storchaka (Anthony Sottile) in branch '2.7':
bpo-32539: Fix OSError for os.listdir() for extended-length paths on Windows (#5169)
https://github.com/python/cpython/commit/27f32e938ff51fd5d90a29abbbabc6b81d156f33
msg310018 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-15 21:40
Thank you Anthony for your patch!
History
Date User Action Args
2018-01-15 21:40:26serhiy.storchakasetstatus: open -> closed
resolution: fixed
messages: + msg310018

stage: patch review -> resolved
2018-01-15 21:39:07serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg310017
2018-01-12 17:38:24eryksunsettype: behavior
stage: patch review
2018-01-12 17:34:16eryksunsetnosy: + eryksun

messages: + msg309866
stage: patch review -> (no value)
2018-01-12 17:27:34Anthony Sottilesetkeywords: + patch
stage: patch review
pull_requests: + pull_request5023
2018-01-12 16:56:45Anthony Sottilecreate