_mkstemp_inner assumes that an access denied error means that it
has generated a filename that matches an existing foldername.
However, in the case of a folder for which you don't have permissions to
create a file, this means it will loop thru the maximum possible number of files.
This causes it to hang for several seconds and eventually return a bogus
FileExistsError.

Similar behaviour exists in 2.7.7, but it throws an IOError instead.

http://bugs.python.org/issue18849 seems to be where this was introduced.

changeset 035b61b52caa has this:-

             return (fd, _os.path.abspath(file))
         except FileExistsError:
             continue    # try again
+        except PermissionError:
+            # This exception is thrown when a directory with the chosen name
+            # already exists on windows.
+            if _os.name == 'nt':
+                continue
+            else:
+                raise
 
     raise FileExistsError(_errno.EEXIST,
                           "No usable temporary file name found")

Could we simply set a flag saying it's a PermissionError and then raise the appropriate PermissionError or FileExistsError at the end of the loop?

What exceptions are raised on Windows when try to open a file in bad directory?

open('foo').close()
open('foo/bar') # what raised?
open('nonexistent/bar') # what raised?

If raised the same exceptions as on Linux, then perhaps the following patch fixes this issue.

And what returns os.access for writable directories and non-existent files?

os.mkdir('dir')
os.access('dir', os.W_OK)          # what returns?
os.access('nonexistent', os.W_OK)  # what returns or raises?

Thank you Mark. Could you please make first test in msg236028 (when first part of the path is a file, not a directory)?

Is there a difference if you do open(..., 'w')? It's a different enough operation that it may have a different error.

Is there a difference if you do open(..., 'w')? It's a different enough operation that it may have a different error.

Oh, yes, I forgot the 'w' mode.

Mark, could you please run following test on Windows?

import os
open('foo', 'wb').close()
flags = os.O_RDWR | os.O_CREAT | os.O_EXCL | getattr(os, 'O_NOFOLLOW', 0) | getattr(os, 'O_BINARY', 0)
os.open('foo/bar', flags, 0o600)          # what raised?
os.open('nonexistent/bar', flags, 0o600)  # what raised?

Great. There is only one difference between Windows and Linux, but it affects only error type in tests. Here is a patch with updated test. It should now work on Windows.

os.access doesn't check filesystem permissions, so the patch will not catch the condition that creates the problem.

It is best that we can do. How else we can check filesystem permissions? Only trying to create a file, But we just tried this and it failed.

It doesn't actually do anything, so why do it at all? In order to distinguish why it failed, you might try checking if the file actually exists, and if it is a folder.

And, just to be clear to Serhiy who I know doesn't use Windows,
os.access really is a worthless function in its present form: worse,
even, because it can be misleading. I have a long-standing patch to
convert it to use AccessCheck but I've never quite had the guts to
commit it because I fear the breakage would be too great.

The main issue is not tempfile raises a FileExistsError, but that it hangs for several seconds (for example if the temp dir doesn't exist). The patch allows to fail early and try other temp dir.

os.access() is not enough, we can add os.path.isdir(). Could you please run tests on patched Python on Windows and say what tests are failed?

The feedback here https://mail.python.org/pipermail/python-dev/2011-May/111530.html seems positive. It references bpo-2528 which is still open but strikes me as the way forward. Why don't we go for it and nail this issue once and for all?

Added os.path.isdir().

Could anybody please run tests on Windows?

======================================================================
ERROR: test_read_only_directory (test.test_tempfile.TestMkdtemp)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 267, in _inside_empty_temp_dir
    yield
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 286, in test_read_only_directory
    self.skipTest("can't set the directory read-only")
  File "C:\Work\Projects\cpython\lib\unittest\case.py", line 645, in skipTest
    raise SkipTest(reason)
unittest.case.SkipTest: can't set the directory read-only

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 289, in test_read_only_directory
    self.assertEqual(os.listdir(tempfile.tempdir), [])
  File "C:\Work\Projects\cpython\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 269, in _inside_empty_temp_dir
    support.rmtree(dir)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 374, in rmtree
    _rmtree(path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 354, in _rmtree
    _waitfor(os.rmdir, path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 301, in _waitfor
    func(pathname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Gustav\\AppData\\Local\\Temp\\tmpe53kiky0'

======================================================================
ERROR: test_read_only_directory (test.test_tempfile.TestMkstempInner)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 267, in _inside_empty_temp_dir
    yield
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 286, in test_read_only_directory
    self.skipTest("can't set the directory read-only")
  File "C:\Work\Projects\cpython\lib\unittest\case.py", line 645, in skipTest
    raise SkipTest(reason)
unittest.case.SkipTest: can't set the directory read-only

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 289, in test_read_only_directory
    self.assertEqual(os.listdir(tempfile.tempdir), [])
  File "C:\Work\Projects\cpython\lib\contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Work\Projects\cpython\lib\test\test_tempfile.py", line 269, in _inside_empty_temp_dir
    support.rmtree(dir)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 374, in rmtree
    _rmtree(path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 354, in _rmtree
    _waitfor(os.rmdir, path)
  File "C:\Work\Projects\cpython\lib\test\support\__init__.py", line 301, in _waitfor
    func(pathname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\Gustav\\AppData\\Local\\Temp\\tmp0qwkkr7l'

Thank you Paul. What with updated patch?

Works fine with the new patch:

.\rt.bat -x64 -q test_tempfile

C:\Work\Projects\cpython\PCbuild>"C:\Work\Projects\cpython\PCbuild\amd64\python.exe" -Wd -E -bb "C:\Work\Projects\cpython\PCbuild\..\lib\test\regrtest.py" test_tempfile

[1/1] test_tempfile
1 test OK.

New changeset 63f0ae6e218a by Serhiy Storchaka in branch '2.7':
Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again
https://hg.python.org/cpython/rev/63f0ae6e218a

New changeset 3a387854d106 by Serhiy Storchaka in branch '3.4':
Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again
https://hg.python.org/cpython/rev/3a387854d106

New changeset 1134198e23bd by Serhiy Storchaka in branch 'default':
Issue bpo-22107: tempfile.gettempdir() and tempfile.mkdtemp() now try again
https://hg.python.org/cpython/rev/1134198e23bd

Unfortunately the patch doesn't fix original issue and looks as this can't be fixed until os.access will be more useful on Windows. But it fixes several related issues. mkstemp() now fails early if parent directory doesn't exist or is a file. gettempdir() and mkdtemp() now try again in case of collision on Windows as well as on Unix. I hope that fixing os.access will automatically fix original issue.

Thanks Mark and Paul for testing on Windows. Thanks Tim for explaining issue with os.access.

My reluctance to commit the os.access patch is because it will cause
such a behaviour change in a function which has been pretty stable for a
long while. It'll certainly be more correct, but at the undoubted
expense of breaking someone's long-working code.

I'm not sure I follow. Isn't the point of this patch to try again in certain cases of a PermissionError, where currently the code breaks out of the loop early? How can the result be worse than the current behaviour? Suerly sometimes (maybe not always) it works now where it previously failed, but that's a good thing?

Shouldn't it be checking whether file (or filename) is a directory [1]? For example:

    except PermissionError:
        # This exception is thrown when a directory with
        # the chosen name already exists on windows.
        if _os.name == 'nt' and _os.path.isdir(file):
            continue
        # If the directory allows write access, continue 
        # trying names. On Windows, currently this test 
        # doesn't work. The implementation assumes all 
        # directories allow write access, but it really 
        # depends on the directory's discretionary and 
        # system access control lists (DACL & SACL).
        elif _os.access(dir, _os.W_OK):
            continue
        else:
            raise

[1]: Windows sets the last error to ERROR_ACCESS_DENIED when a system
call returns the NTSTATUS code STATUS_FILE_IS_A_DIRECTORY. This
status code is returned by NtCreateFile and NtOpenFile when the
target file is a directory, but the caller asked for anything
but a directory (i.e. CreateOptions contains
FILE_NON_DIRECTORY_FILE).

There is a risk of race condition. One process can create a directory file, then other process fails to create a file with the same name file, then the first process removes directory file, then the second process handles PermissionError. The same is possible also with threads.

I just experienced the described problem using Python 3.6.0 (Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32), but i do not understand the current status of this issue: On the one hand it is marked as 'open', which seems to imply that is still not resolved. On the other hand the resolution was set to 'fixed' in May 2015. Should i open a new issue for Python 3.6?

This issue persists as of today (March 2019), in Python 3.7.2 (64 bit) running on Windows 10. I gather from the comments that fixing it is no trivial matter, although I don't fully understand why. The hang of "several seconds" that was originally described is at least 30 seconds on that platform -- I'm not sure when it would clear, as I didn't have the patience to wait it out.

https://stackoverflow.com/questions/55109076/python-tempfile-temporaryfile-hangs-on-windows-when-no-write-privilege

It seems to me that a genuine naming collision would be pretty rare -- at least for the (fairly common) use case I'm dealing with, trying to check where the directory is writeable or not. Would it make sense to at least set a default number of collision-retries that is significantly lower? Say, between 3 and 10, instead of the system max?

I stand by the patch file I previously submitted on 2016-05-04. A more detailed analysis / description of my reasoning follows.

Change 1 in _get_default_tempdir:
A PermissionError is thrown on Windows if you attempt to create a file whose filename matches an existing directory. As the code currently stands, the if statement checks whether the proposed file's parent directory is a directory (which it is, or a FileNotFoundError would have been thrown), instead of whether the proposed filename conflicts with an existing directory. The edited expression is really a typo that, in the context of the code block, always evaluates True.
Here’s what we’re now saying in the if block: when a PermissionError is raised, if we’re on Windows (the only currently supported platform to throw nonsense errors at us) AND the filename we chose simply conflicts with an existing directory AND we supposedly have write access to the parent directory, then we were just unlucky with the chosen name and should try again in the same parent directory. (I say supposedly because Windows seems to erroneously report True on this check, even when we don’t have write access. I wouldn’t be surprised if this last check does something useful in certain contexts, I just don’t know what they are.)

Change 2 in _mkstemp_inner:
Same as above for Change 1. While _get_default_tempdir uses this code block to make sure the system tempdir is really writable, and _mkstemp_inner does it so that a file descriptor can be returned, the result and arguments are the same.

Change 3 in mkdtemp:
For _get_default_tempdir and _mkstemp_inner, the blocks of code in question are creating temporary files. As such, they need to handle the oddball case for Windows where attempts to create a file with a filename which conflicts with an existing directory name result in a PermissionError. The same block of error handling code is copied in mkdtemp, even though this function never tries to create a file – it only tries to create a directory. As such, in the case that we try to create a directory with a name that already exists (whether as a file or a directory), we wouldn't be dealing with a PermissionError, we'd have a FileExistsError, which is already handled in mkdtemp by the preceding lines. The only way I’ve seen a PermissionError crop up for a call to mkdir on Windows is if the user doesn’t have permission to create filesystem objects in the parent directory. This is the intended usage of a PermissionError, so no special handling needed is required. Remember, a PermissionError shouldn’t happen if mkdtemp is called without a dir kwarg, because the _sanitize_params will invoke _get_default_tempdir, which will check to ensure that the parent directory is writable. As such, this block of code was superfluous, and the patch should not raise PermissionError in user code where it previously was caught.

Series of operations needed to answer the questions os.access is not answering on windows:

bool CanAccessFolder( LPCTSTR folderName, DWORD genericAccessRights )
{
    bool bRet = false;
    DWORD length = 0;
    if (!::GetFileSecurity( folderName, OWNER_SECURITY_INFORMATION | GROUP_SECURITY_INFORMATION 
            | DACL_SECURITY_INFORMATION, NULL, NULL, &length ) && 
            ERROR_INSUFFICIENT_BUFFER == ::GetLastError()) {
        PSECURITY_DESCRIPTOR security = static_cast< PSECURITY_DESCRIPTOR >( ::malloc( length ) );
        if (security && ::GetFileSecurity( folderName, OWNER_SECURITY_INFORMATION | GROUP_SECURITY_INFORMATION
                            | DACL_SECURITY_INFORMATION, security, length, &length )) {
            HANDLE hToken = NULL;
            if (::OpenProcessToken( ::GetCurrentProcess(), TOKEN_IMPERSONATE | TOKEN_QUERY | 
                    TOKEN_DUPLICATE | STANDARD_RIGHTS_READ, &hToken )) {
                HANDLE hImpersonatedToken = NULL;
                if (::DuplicateToken( hToken, SecurityImpersonation, &hImpersonatedToken )) {
                    GENERIC_MAPPING mapping = { 0xFFFFFFFF };
                    PRIVILEGE_SET privileges = { 0 };
                    DWORD grantedAccess = 0, privilegesLength = sizeof( privileges );
                    BOOL result = FALSE;
 
                    mapping.GenericRead = FILE_GENERIC_READ;
                    mapping.GenericWrite = FILE_GENERIC_WRITE;
                    mapping.GenericExecute = FILE_GENERIC_EXECUTE;
                    mapping.GenericAll = FILE_ALL_ACCESS;
 
                    ::MapGenericMask( &genericAccessRights, &mapping );
                    if (::AccessCheck( security, hImpersonatedToken, genericAccessRights, 
                            &mapping, &privileges, &privilegesLength, &grantedAccess, &result )) {
                        bRet = (result == TRUE);
                    }
                    ::CloseHandle( hImpersonatedToken );
                }
                ::CloseHandle( hToken );
            }
            ::free( security );
        }
    }
 
    return bRet;
}

CanAccessFolder is incomplete for the following reasons:

(1) It doesn't account for SeBackupPrivilege (read and execute access) and SeRestorePrivilege (write and delete access). If a create or open call requests backup semantics, these two privileges are checked by the I/O Manager in order to grant access before the request is sent to the filesystem device stack. That said, our os.open() call doesn't use O_OBTAIN_DIR (0x2000), an undocumented flag that allows opening a directory by requesting backup semantics. If it did use this undocumented flag, there would actually be no problem to solve since we would get FileExistsError instead of PermissionError.

(2) It doesn't check the parent directory in case FILE_READ_ATTRIBUTES access (required for GENERIC_READ) or DELETE access (required for GENERIC_ALL) are either not granted or denied to the user for the target directory. These rights are granted by NT filesystem policy if FILE_READ_DATA and FILE_DELETE_CHILD access are granted to the parent directory, respectively. This is a general concern that's not relevant here since we only need to check the directory for write access (i.e. FILE_ADD_FILE).

(3) It doesn't get the LABEL_SECURITY_INFORMATION from the file security, in order to check for no-read-up, no-write-up, and no-execute-up mandatory access. Adding files to a directory is disallowed if its mandatory label denies write-up access and its integrity level is higher than the caller's (e.g. the caller is a standard user at medium integrity level, and the directory is at high or system integrity level).

(4) It doesn't check effective access via OpenThreadToken, in case the thread is impersonating.

(5) It cannot take into account access granted or denied by filesystem filter drivers such as antimalware programs. For this, we need to actually try to open the directory with the requested access via CreateFile. We're granted the required access if CreateFile succeeds, or if it fails with a sharing violation (i.e. winerror 32). A sharing violation isn't an issue since in practice adding a file to a directory is internal to a filesystem; it doesn't count against shared data access.

Attempting to answer the question "did this open call fail because the path was a directory" by implication from "do we think we ought to be able to write a file to this directory" is IMO doomed. There's no reliable way to determine whether one should be able to write to a location, short of trying to write to it. There isn't in general and there especially isn't on Windows, for the reasons discussed by Eryk and more.

I believe Billy's patch is an improvement over what we have, in as much as it's specifically checking for the condition we care about to work around a shortcoming of Windows error reporting. I would further remove the use of os.access, which does nothing useful (it reflects the read-only file attribute, which no-one uses, and which doesn't exist for directories anyway).

Yes, there is a race condition if the directory goes away between use and check, but it seems vanishingly unlikely this could happen by accident, and it could only happen on purpose if an attacker can guess the random filenames in which case they already have worse attacks than just making mkstemp fail. In general failure is a much better outcome than hanging for hours on end.

We may be overthinking this. Maybe it is OK to treat all permission errors as maybe-file-exists errors, like bpo-18849 originally did, and just retry without trying to pick apart the entrails.

...just not quite as many as 2147483647 times. Given how unlikely an accidental filename clash is in the first place, I'm thinking a more realistic number of retries might be something like, 2.

os.TMP_MAX probably isn't a good choice in any case, as it indicates the number of names C's tmpnam() can come up with, which (a) is unrelated to the number of names _RandomNameSequence can come up with and (b) we have no particular reason to try to completely exhaust anyway.

i would like to point out that the primary reason any of this nonsense exists is because of short filename restrictions.

i've replaces nearly all of my temp file creation code in all of my project to return os.urandom(32).hex() ... which is reliable secure, fast, and doesn't require any fileops

sure, it doesn't work on 8.3 limited systems, but maybe the NamedTemp code should check that *first*.... and *if* it's OK to use long names... just use them, otherwise revert to existing behavior

This is the fist of what I'm using:
https://gist.github.com/earonesty/a052ce176e99d5a659472d0dab6ea361

Seems OK for my use cases. There's probably issues with relying on __del__ this way. But it solves the Windows close/reopen problem, too.

I can't believe this issue haven't been solved after ten years. And some package really takes tempfile.TemporaryFile() for writability check. Hope Windows users enjoy the 2147483647 loops.

Maybe there really is a reason for PathYetAnotherMakeUniqueName?

Ran into this issue yesterday when I tried using a module that used Numba on Windows. Somebody has probably already mentioned this, but I'm not reading this whole thread. The root of the problem is this comment in posixmodule.c:

(Directories cannot be read-only on Windows.)

This is NOT true. While there is no exact (single) equivalent, Windows has multiple permissions that make directories effectively read-only. In this case the permission that is lacking is FILE_ADD_FILE, which allows files to be created in that directory (an alias of FILE_WRITE_DATA, which is the effect it has when applied to files). Related is FILE_ADD_SUBDIRECTORY (an alias of FILE_APPEND_DATA).

After a bit of searching, I found a straightforward solution in a 2004 (cough 10 years before the problem appeared in Python cough) post by old Chen. Here is a hacky workaround I made for my own use. It probably isn't sufficiently robust for production code, but it gets the job done until Python properly fixes it:

import win32api
import win32
import win32file
import pywintypes

orig_access = os.access
def my_access(path, mode, *, dir_fd=None, effective_ids=False, follow_symlinks=True):
	res = orig_access(path, mode, dir_fd = dir_fd, effective_ids = effective_ids, follow_symlinks = follow_symlinks)
	if not res or not mode & os.W_OK or not os.path.isdir(path):
		return res
	
	try:
		hdl = win32file.CreateFile(
			str(path), 
			2, # FILE_ADD_FILE
			win32file.FILE_SHARE_READ | win32file.FILE_SHARE_WRITE | win32file.FILE_SHARE_DELETE, 
			None, 
			win32file.OPEN_EXISTING, 
			win32file.FILE_FLAG_BACKUP_SEMANTICS, 
			None,
		)
		win32file.CloseHandle(hdl)
		
	except pywintypes.error as e:
		if e.winerror == 5: # ERROR_ACCESS_DENIED
			return False
		
	except Exception as e:
		pass
	
	return True

os.access = my_access```

Changing os.access to detect a directory (which it already does) and testing for FILE_ADD_FILE access as shown by @TheRealQuantam seems reasonable. If someone wants to work up a PR, I'm sure we could move that forward.

I'm not sure whether it would unblock the original issue, but only because I also haven't re-read the entire thread.

For an official fix it would probably be best to make os.access test FILE_ADD_FILE | FILE_ADD_SUBDIRECTORY (6), as that's the more conservative option. That is, due to there being multiple permissions in Windows it's impossible for os.access to be 100% accurate in the edge case where a directory has add file but not add subdirectory permissions (or vice-versa); in such cases I'd think it's better to report that the directory is not writable even though files or directories can be created than report that it is writable only for attempts to create things to then fail.

I can confirm that at least in the case of Numba (which used tempfile to create temporary files) my workaround does in fact fix the issue with _mkstemp_inner (the topic of this issue). The attempt to create a temp file in a non-writable directory fails immediately for access denied, and Numba falls back to using the user's temp directory (the proper solution).