Title: Inconsistent os.stat behavior for directory with Access Denied
Type: behavior Stage:
Components: Windows Versions: Python 3.8, Python 3.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: CrouZ, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-11-16 18:14 by CrouZ, last changed 2021-01-09 14:44 by CrouZ.

Messages (4)
msg356763 - (view) Author: (CrouZ) Date: 2019-11-16 18:14
After upgrading some scripts from Python 2.7 to 3.7 in Windows 10, I got different behavior that seems to be caused by inconsistent behavior for os.stat in Python 3.7.

Python 2.7:
>>> os.stat("D:\\System Volume Information")
nt.stat_result ...
>>> os.stat("D:\\System Volume Information\\")
nt.stat_result ... (same as previous call)

Python 3.7:
>>> os.stat("D:\\System Volume Information")
os.stat_result ...
>>> os.stat("D:\\System Volume Information\\")
Traceback ...
PermissionError: [WinError 5] Access is denied: 'D:\\System Volume Information\\'

What I really do is calling:
>>> os.path.exists("D:\\System Volume Information\\")
False (Unexpected and inconsistent. I expect the return value to be True.)

Behavior for other calls:
>>> os.path.exists("D:\\System Volume Information")
True (OK)
>>> os.path.isdir("D:\\System Volume Information\\")
True (OK, but according to the documentation "Return True if path is an existing directory." where 'existing' links to os.path.exists, which returns False)

The closest issue I could find was Issue28075 which has already been fixed.
msg356885 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-11-18 17:03
I haven't debugged it, but I'm guessing we're not handling the trailing slash properly when falling back to asking the parent directory for information.

Looking at the actual stat result for "C:\System Volume Information" vs. "C:\Windows", most of the information is missing in the first case, which should mean that we can't access it but we know it's a directory because the entry in C:\ said so.
msg356915 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-11-18 21:53
In attributes_from_dir() in Modules/posixmodule.c, a trailing backslash or slash should be stripped from the lpFileName parameter of FindFirstFileW. Otherwise the call opens the directory via NtOpenFile, instead of opening its parent directory. Even if opening the directory is successful, which we don't expect in this case, FindFirstFileW forcibly fails the call with ERROR_FILE_NOT_FOUND (2) because it expects a filename filter (e.g. "*") for the internal NtQueryDirectoryFile[Ex] system call.

Care needs to be taken to not strip the trailing slash of the root directory of a DOS drive because that creates a drive-relative path (e.g. "C:"). It is expected that FindFirstFileW will fail for the root of a DOS drive or UNC share, since there's no parent directory to open.


"System Volume Information" explicitly grants access only to the SYSTEM account. Implicitly we have read-attributes access to this directory because we have read-data (i.e. list-directory) access to the root directory. Great, but even for 0 desired access, CreateFileW requests both read-attributes and synchronize access, even for overlapped I/O (i.e. kernel File objects created by CreateFileW can always be waited on). So even an elevated administrator normally can't open this directory to query information. However, backup and restore privileges are in effect when an open requests backup semantics, which we already do. We could extend os.stat to temporarily enable SeBackupPrivilege if the caller has it. 

It's also possible to open the directory with a native NtOpenFile or NtCreateFile system call, without the FILE_SYNCHRONOUS_IO_NONALERT option and without requesting SYNCHRONIZE access -- i.e. the File object will be asynchronous and not waitable.
msg384729 - (view) Author: (CrouZ) Date: 2021-01-09 14:44
The problem exists in Python 3.8 as well, with the difference that ``os.path.isdir("D:\\System Volume Information\\")`` also returns False.

Tested with Python 3.8.2 and 3.8.7.
Date User Action Args
2021-01-09 14:44:44CrouZsetmessages: + msg384729
versions: + Python 3.8
2019-11-18 21:53:30eryksunsetnosy: + eryksun
messages: + msg356915
2019-11-18 17:03:58steve.dowersetmessages: + msg356885
2019-11-16 18:14:54CrouZcreate