classification
Title: os.listdir() got permission error in Python3.6 but it's fine in Python2.7
Type: behavior Stage: resolved
Components: Windows Versions: Python 3.6
process
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: Ryan_D@163.com, eryksun, paul.moore, steve.dower, steven.daprano, tim.golden, zach.ware
Priority: normal Keywords:

Created on 2019-04-17 00:46 by Ryan_D@163.com, last changed 2021-03-16 04:03 by eryksun. This issue is now closed.

Files
File name Uploaded Description Edit
X1ONx.png Ryan_D@163.com, 2019-04-17 00:46 screenshot when problem occurred
Messages (12)
msg340373 - (view) Author: Ryan (Ryan_D@163.com) Date: 2019-04-17 00:46
My script need scan a netdisk directory to get the content of it. I use os.listdir() method for an easy implement, then I got permission error when executing in Python 3.x, but the same code is working fine in Python 2.7,I attached a screenshot for explaining the problem.
msg340375 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-04-17 01:18
Please don't post screen shots of text, copy and paste the text as text.

Screen shots are hostile to the blind and visually impaired as screen-readers don't work with them. They make it impossible for us to copy your code to run it ourselves, and text in screen shots cannot be searched for.


As this is a permission error, the chances are very strong that it is not a bug but an actual OS permission error. You need to eliminate site-specific factors:

Are you running the two different versions of Python using the same user account with the same access controls? If the two versions of Python run with different permissions, that may explain why you get a permission error.

What are the permssions on the file you are trying to access? Could there be any additional OS-level or network drive ACLs?

Being a networked drive, is it possible that it is a transient network glitch being wrongly reported by Windows as a permission error? In other words, do you get the same error every time, or was it just a one-off?
msg340378 - (view) Author: Ryan (Ryan_D@163.com) Date: 2019-04-17 02:22
Hi Steven,
Thanks for your reply, I paste the output I executed just now as below.
You can see that the both version of Python are running in the same shell with the same permission. And the frequency of this problem is always happen. BTW, the network disk is an intranet path of our company, we have the read permission of it.

PS D:\workspace> python
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir(r'L:\Temp')
['manager.txt', 'alps-trunk-m0.tk.log', '2015102914.LOG', 'log', 'alps-trunk-m0.tk-k55mv1_64_om_c2k6m_eng-2015111215.LOG
', 'logof', '1', 'dsp_1_ltg_n.bin', 'dsp_1_lwg_n.bin', 'modem_1_ltg_n.img', 'modem_1_lwg_n.img', 'kernel_symbols', 'k55v
1_64_om_c2k6m_clang.log', 'k55v1_64_om_c2k6m_android.log', 'k50v1_64_om_c2k6m_android.log', 'collectBuildLog.sh', 'Utili
tySpotlight', 'k55v1_bsp_android.log', 'specialowner.pm', 'DBbuild_GIT.pl', 'daily_out-k37mv1_basic', 'mtkall_new.txt',
'Visa', 'insert_err_command_file.txt', 'test', 'DB_X_remake.list', 'aosp_img.txt', 'Jola', '04272', 'alps_dailybuild.ini
', 'Luca', 'android-security-bulletin', 'michael', '2018073115.LOG', 'LOG~1', 'mtk-db.pl', 'MCD_k79v1_64_android.log', '
build-full_k79v1_64.ninja']
>>> exit()
PS D:\workspace> python3
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir(r'L:\Temp')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 5] 拒绝访问。: 'L:\\Temp'
>>> exit()
PS D:\workspace>
msg340379 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-04-17 02:38
Thanks Ryan.

> PermissionError: [WinError 5] 拒绝访问。: 'L:\\Temp'

Can you translate the error message into English for us please?
msg340380 - (view) Author: Ryan (Ryan_D@163.com) Date: 2019-04-17 03:10
Hi Steven,
> PermissionError: [WinError 5] 拒绝访问。: 'L:\\Temp'
to EN
> PermissionError: [WinError 5] Access denied.: 'L:\\Temp'
msg340389 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-04-17 10:51
Try os.listdir(u'L:\\Temp') in Python 2. Using a unicode string should result in the same WINAPI FindFirstFileW call as in Python 3.

You're running `python3`, but the standard installation doesn't include "python3.exe". Please show the output of `(get-command python3).source` in PowerShell.

Compare the security context (user, groups, privileges) by running subprocess.call('whoami /all /fo list') from both Python 2 and 3.
msg340394 - (view) Author: Ryan (Ryan_D@163.com) Date: 2019-04-17 13:15
I ran "python3" because I rename the execute file for distincting with python2 exe.
PS D:\workspace> (get-command python3).source
C:\Python36_64\python3.exe

There is no problem for my dev environment, it's an obviously different output for the same API call between python2.7 and python3.6.
not only listdir(), but also os.walk() has the same problem.
msg340397 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-04-17 13:53
Did you try os.listdir(u'L:\\Temp') in Python 2?
msg340463 - (view) Author: Ryan (Ryan_D@163.com) Date: 2019-04-18 02:33
PS D:\workspace> python
Python 2.7.15 (v2.7.15:ca079a3ea3, Apr 30 2018, 16:30:26) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir(u'L:\\Temp')
[u'manager.txt', u'alps-trunk-m0.tk.log', u'2015102914.LOG', u'log', u'alps-trunk-m0.tk-k55mv1_64_om_c2k6m_eng-201511121
5.LOG', u'logof', u'1', u'dsp_1_ltg_n.bin', u'dsp_1_lwg_n.bin', u'modem_1_ltg_n.img', u'modem_1_lwg_n.img', u'kernel_sym
bols', u'k55v1_64_om_c2k6m_clang.log', u'k55v1_64_om_c2k6m_android.log', u'k50v1_64_om_c2k6m_android.log', u'collectBuil
dLog.sh', u'UtilitySpotlight', u'k55v1_bsp_android.log', u'specialowner.pm', u'DBbuild_GIT.pl', u'daily_out-k37mv1_basic
', u'mtkall_new.txt', u'Visa', u'insert_err_command_file.txt', u'test', u'DB_X_remake.list', u'aosp_img.txt', u'Jola', u
'04272', u'alps_dailybuild.ini', u'Luca', u'android-security-bulletin', u'michael', u'2018073115.LOG', u'LOG~1', u'mtk-d
b.pl', u'MCD_k79v1_64_android.log', u'build-full_k79v1_64.ninja']
>>> exit()
PS D:\workspace>
PS D:\workspace>
PS D:\workspace> python3
Python 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir(u'L:\\Temp')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
PermissionError: [WinError 5] Access denied.: 'L:\\Temp'
>>> exit()
PS D:\workspace>
msg340466 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2019-04-18 03:21
This is weird.

Are you able to tell us anything about how L: is mounted or what it's pointing at? Do you know if it's SMB or similar, or if there are symlinks or similar?
msg340486 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2019-04-18 13:17
For me in Windows 10, os.listdir(u"C:\\Temp") uses the same sequence of WINAPI and NT system calls in 2.7 and 3.6.

Setup the query for all files ("*"), and return the first matching entry:

    FindFirstFileW
        FindFirstFileExW

            NtOpenFile(
                ObjectAttributes=(
                    ObjectName="\??\C:\Temp\",
                    Attributes=OBJ_CASE_INSENSITIVE)
                DesiredAccess=(
                    SYNCHRONIZE | 
                    FILE_LIST_DIRECTORY),
                ShareAccess=(
                    FILE_SHARE_READ | 
                    FILE_SHARE_WRITE | 
                    FILE_SHARE_DELETE),
                OpenOptions=(
                    FILE_DIRECTORY_FILE | 
                    FILE_SYNCHRONOUS_IO_NONALERT |
                    FILE_OPEN_FOR_BACKUP_INTENT))

            NtQuerDirectoryFileEx(
                FileInformationClass=FileBothDirectoryInformation, 
                QueryFlags=SL_RETURN_SINGLE_ENTRY,
                FileName="*")

Repeat for the remaining entries:

    FindNextFileW

        Return the next entry from a 4 KiB buffer. If the buffer is
        empty, refill it with the following call, until the query is
        exhausted (i.e. STATUS_NO_MORE_FILES):

            NtQuerDirectoryFileEx(
                FileInformationClass=FileBothDirectoryInformation,
                Length=4096)

Unfortunately, many NT status codes map to ERROR_ACCESS_DENIED (5). If it's STATUS_ACCESS_DENIED (0xC0000022), then probably NtOpenFile failed. Try checking the last NT status value with ctypes. For example:

    >>> import os, ctypes
    >>> ntdll = ctypes.WinDLL('ntdll')
    >>> ntdll.RtlGetLastNtStatus.restype = ctypes.c_ulong
    >>> try:
    ...     os.listdir(u'C:\\Users\\Administrator')
    ... except:
    ...     print(hex(ntdll.RtlGetLastNtStatus()))
    ...
    0xc0000022

Do you have any anti-malware programs running? If so, try disabling them. They can hook APIs or attach to the I/O stack with a filter driver (e.g. to deny an I/O request that's disallowed by a rule). If disabling your anti-malware software allows access, then you'll need to set a rule to grant access to Python 3.
msg388808 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2021-03-16 04:03
I'm certain this is a third-party problem -- something like AppLocker or an anti-malware program that's limiting filesystem access. It's not a bug in Python, which simply uses FindFirstFileW() and FindNextFileW() in the normal way.
History
Date User Action Args
2021-03-16 04:03:04eryksunsetstatus: open -> closed
resolution: third party
messages: + msg388808

stage: resolved
2019-04-18 13:17:19eryksunsetmessages: + msg340486
2019-04-18 03:21:12steve.dowersetmessages: + msg340466
2019-04-18 02:33:05Ryan_D@163.comsetmessages: + msg340463
2019-04-17 13:53:47eryksunsetmessages: + msg340397
2019-04-17 13:15:40Ryan_D@163.comsetmessages: + msg340394
2019-04-17 10:51:25eryksunsetnosy: + eryksun
messages: + msg340389
2019-04-17 03:10:10Ryan_D@163.comsetmessages: + msg340380
2019-04-17 02:38:35steven.dapranosetmessages: + msg340379
2019-04-17 02:22:04Ryan_D@163.comsetmessages: + msg340378
2019-04-17 01:18:46steven.dapranosetnosy: + steven.daprano
messages: + msg340375
2019-04-17 00:46:40Ryan_D@163.comcreate