On Unix, files (unless specifically locked) can be renamed and deleted while file descriptors for the file remain open.  The file descriptors remain valid even after deletion of the file.

On Windows this is not possible for files opened using io.open() or os.open().  However, by using the FILE_SHARE_DELETE flag in CreateFile() one can get Unix-like behaviour.

Unfortunately, FILE_SHARE_DELETE is only available through the Win32 API, not through the CRT.  Also, Issue #14243 concerns the fact that on Windows temporary files cannot be reopened unless one uses the FILE_SHARE_DELETE flag.  One can only reopen a file by using a share mode that is at least as permissive as the share mode for all the currently open handles.

The attached patch adds a module "share" (bad name?) with functions share.open() and share.os_open() which act as substitutes for io.open() and os.open().  These by default use FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE as the share mode.  (io.open() and os.open() use FILE_SHARE_READ | FILE_SHARE_WRITE instead.)

To run the full regression test suite with builtins.open(), io.open() and os.open() monkey patched to use these replacements you can do

    python -m test.test_share --regrtest

Nothing seems to break.

Having the same semantics on both Unix and Windows with regard to validity of file handle after a file was deleted would be a very nice to have. How could we progress this?

I'm adding Martin and Antoine to cc list.

Actually, it is not quite the same semantics as Unix.

After you delete the the file you cannot create a file of the same name or delete the directory which contains it until the handle has been closed.

However, one can work around that by moving the file somewhere else (like the root directory) before deleting it.

I don't understand whether you are proposing to include the patch into Python as-is; if so, I'm -1 on it for two formal reasons: a) the standard library shouldn't monkey patch itself, and b) OS interfacing should be implemented in C.

That said, having maximum sharing when opnening files sounds fine to me.

> I don't understand whether you are proposing to include the patch into Python as-is;

I think Richard is well aware of the constraints you specify and current patch was meant as a proof of concept; to show that all tests pass with such a change. Of course that's only my belief and we shall see what Richard has to say.

> That said, having maximum sharing when opnening files sounds fine to me.

Good to hear. However I started to wonder if we are ready for all consequences of this. For example taking into account what Richard noted in http://bugs.python.org/issue14243, specifically:

> Unfortunately using O_TEMPORARY is the only way allowed by msvcrt to
> get FILE_SHARE_DELETE, even though it also has the orthogonal effect
> of unlinking the file when all handles are closed.

forces programs which would like to open a file being opened at the same time by Python code (by means of built-in open() or os.open() with default arguments) to either use O_TEMPORARY when using msvcrt or to go low level and use CreateFile() Win32 API function with FILE_SHARE_DELETE flag. Are we ok with it?

Am 14.03.13 03:31, schrieb Piotr Dobrogost:
> forces programs which would like to open a file being opened at the
> same time by Python code (by means of built-in open() or os.open()
> with default arguments) to either use O_TEMPORARY when using msvcrt
> or to go low level and use CreateFile() Win32 API function with
> FILE_SHARE_DELETE flag. Are we ok with it?

That's why I was asking for an actual patch. The proposed change may
well not be implementable. If os.open continues to create CRT handles,
a way needs to be found to get a CRT handle that as the
FILE_SHARE_DELETE bit set.

An alternative approach could be that os.open stops creating CRT
handles, and directly uses OS handles. The problem with that is that
stdin/stdout/stderr would stop being 0/1/2, which is not acceptable.
An alternative solution to that could be that we introduce a notion
of "python io handles", parallel, but indepedendent from CRT handles.
And so on.

On 14/03/2013 1:00pm, Martin v. Löwis wrote:
> That's why I was asking for an actual patch. The proposed change may
> well not be implementable. If os.open continues to create CRT handles,
> a way needs to be found to get a CRT handle that as the
> FILE_SHARE_DELETE bit set.

The patch *does* create CRT fds from win32 handles by using 
msvcrt.open_osfhandle().

One other issue is that I do not know of a way to determine the current 
umask without temporarily changing it, causing a thread-race.

In the end I am not sure it is worth the hassle.  (But maybe it would be 
a good idea to add test.support.open() using FILE_SHARE_DELETE and 
test.support.unlink() to make the testsuite more resilient to 
"Permission Denied" errors.)

> An alternative approach could be that os.open stops creating CRT
> handles, and directly uses OS handles. The problem with that is that
> stdin/stdout/stderr would stop being 0/1/2, which is not acceptable.
> An alternative solution to that could be that we introduce a notion
> of "python io handles", parallel, but indepedendent from CRT handles.
> And so on.

http://bugs.python.org/issue12939 has C implementations of _io.WinFileIO 
and _io.openhandle() which are equivalents for _io.FileIO and os.open() 
which use "native" Windows handles.