classification
Title: os.listdir has inconsistent behavior when run on a non-directory
Type: enhancement Stage: resolved
Components: Library (Lib), Windows Versions: Python 3.7
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: eryksun, paul.moore, r.david.murray, raylu, serhiy.storchaka, steve.dower, terry.reedy, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2017-01-25 02:43 by raylu, last changed 2017-06-22 02:50 by terry.reedy. This issue is now closed.

Messages (10)
msg286225 - (view) Author: raylu (raylu) Date: 2017-01-25 02:43
According to
https://github.com/python/cpython/blob/4ed71dae9d23a412f9f73d3a0b1f4be37543e49e/Lib/test/test_unicode_file_functions.py#L106
listdir can sometimes return FileNotFoundError on Windows.

Tangentially related: https://hg.python.org/cpython/rev/385c2ec78f16
msg286258 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-01-25 14:43
Could you please clarify what bug you are reporting?
msg286275 - (view) Author: raylu (raylu) Date: 2017-01-25 20:06
os.listdir should always return NotADirectoryError when run on a regular file. Users shouldn't have to special-case win32/FileNotFoundError.
msg286278 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-01-25 20:21
Blame Windows for this behavior. os.listdir() is just a wrapper around few system calls.

It is expected that functions in the os module can raise different exceptions on different platforms.
msg286279 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-25 20:38
Python appends "\*.*" to the path, so in the case in which the leaf element is a file, FindFirstFile naturally fails with ERROR_PATH_NOT_FOUND.

Python 3.5+ could maybe switch to calling CreateFile to open a handle and GetFileInformationByHandleEx to get the FILE_ID_BOTH_DIR_INFO (added in Vista, so this couldn't be backported to 2.7). It would first have to get the FILE_BASIC_INFO for the file attributes to ensure that the target is a directory. This could also support listing directories by file descriptor on Windows, since the CRT has O_OBTAIN_DIR (0x2000) for opening a directory.
msg286309 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-26 12:36
> FindFirstFile naturally fails with ERROR_PATH_NOT_FOUND

Getting this error actually depends on the file system. I don't see it with NTFS, which returns STATUS_NOT_A_DIRECTORY, which gets translated to ERROR_DIRECTORY. On the other hand, VboxSF (VirtualBox shared folder) returns STATUS_OBJECT_PATH_NOT_FOUND, which gets translated to ERROR_PATH_NOT_FOUND.

Also, there are many possible errors when trying to list a path on a device that isn't managed by a file system (e.g. \\.\PhysicalDrive0). The device can return a status value that FindFirstFile doesn't special case, such as STATUS_UNSUCCESSFUL, STATUS_NOT_FOUND, or STATUS_OBJECT_NAME_INVALID. The NtOpenFile call may even succeed, but then the NtQueryDirectoryFile will probably fail with STATUS_INVALID_DEVICE_REQUEST.
msg286382 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-01-27 19:17
This is a feature-change 'enhancement' issue.  There is no violation of the doc at https://docs.python.org/3/library/os.html#os.listdir and as Raylu noted, the current behavior is accommodated in tests.

I presume that regardless of what system calls are made, FileNotFoundError and anything else could be caught and NotADirectoryError raised instead, with the message containing the Windows error message.  This would only break Windows-specific code that only catches FileNotFoundError and not NotADirectoryError.

I am not sure what our general policy is and whether this should be fixed at this point.  I personally like Python smoothing over OS differences when easily possible.
msg286405 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-01-28 08:46
Python has no specific use for ERROR_PATH_NOT_FOUND (3) and just maps it to FileNotFoundError -- like the CRT maps it to ENOENT. Even if we wanted listdir() to specially handle this error, given that it's implemented via FindFirstFile, there's nothing that can be done after the fact that's not vulnerable to race conditions. 

Also, to reiterate this point, getting ERROR_PATH_NOT_FOUND instead of ERROR_DIRECTORY (267) when the target is a file seems to be pretty uncommon, much more so than I at first thought. I only see it with VirtualBox shared folders. I don't see it with Microsoft file systems such as NTFS, FAT32, or CDFS.
msg296286 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2017-06-18 18:28
Eryk, does your last comment suggest that we close this?
msg296300 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2017-06-18 23:57
Yes, I think this issue should be closed. But for the record I'd like to note a not uncommon case in which listdir() raise FileNotFoundError on Windows.

According to MS-FSA [1], if a request to open a directory resolves to a file, the operation should fail with STATUS_NOT_A_DIRECTORY (see 2.1.5.1, phase 7). That's the scenario I was discussing in previous messages. However, I neglected to discuss what happens when an intermediate path component is not a directory. In theory this case should also fail with STATUS_NOT_A_DIRECTORY (see 2.1.5.1, phase 6). However, in practice MS file systems instead return STATUS_OBJECT_PATH_NOT_FOUND, which becomes ERROR_PATH_NOT_FOUND, for which Python raises FileNotFoundError.

Walking the path to find the reason for the failure shouldn't be attempted because it's subject to race conditions -- e.g. the file that caused the failure may no longer exist. 

[1]: https://msdn.microsoft.com/en-us/library/ff469536.aspx
History
Date User Action Args
2017-06-22 02:50:40terry.reedysetstatus: open -> closed
resolution: not a bug
stage: test needed -> resolved
2017-06-18 23:57:13eryksunsetmessages: + msg296300
2017-06-18 18:28:18terry.reedysetmessages: + msg296286
2017-01-28 08:46:16eryksunsetmessages: + msg286405
2017-01-27 19:17:26terry.reedysetversions: + Python 3.7
nosy: + terry.reedy

messages: + msg286382

type: enhancement
stage: test needed
2017-01-26 12:36:14eryksunsetmessages: + msg286309
2017-01-25 20:38:22eryksunsetnosy: + eryksun
messages: + msg286279
2017-01-25 20:21:18serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg286278
2017-01-25 20:06:22raylusetmessages: + msg286275
2017-01-25 14:43:25r.david.murraysetnosy: + r.david.murray
messages: + msg286258
2017-01-25 02:43:12raylucreate