Currently, removing directories during testing on Windows w/NTFS can causing sporadic failures due to access denied errors.  This is caused by other processes getting a handle to the directory itself (change notifications) or a handle to a file within the directory.  The most notable offender is the Indexing Service.

Most (but not all!) external programs can be configured to ignore the development directory.  For example, the Indexing Service can be disabled or have those directories ignored.  TortoiseSVN is another offender that can exclude directories but each directory needs to be listed separately so it is easy to forgot one.  On my machine I have programs that simply do not have an option to ignore any directories thus causing some grief during testing.

The attached patch to test.support eliminates the need to disable or and ignores to any programs (tested on a Win7-x64 i7-3770K@4.3GHz).

It achieves this by checking for the removal of the directory before returning to the caller.  It performs an exponential backoff timeout loop that amounts to a total of ~1 second in the worst case.  If the directory is not removed from the filesystem by then, it will probably be in error anyway.  However, the loop is seldom executed more than once.

This is a (near) duplicate of issue7443, I think.

> This is a (near) duplicate of issue7443, I think.

Partially so it seems.  However, my testing with Process Monitor (from
sysinterals) shows that most of the access denied errors occur when
removing directories.

The blind rename-remove dance doesn't work when removing entire
directory trees as now the renamed file/directory might be held
pending delete when the removing parent directory.

For some background for newcomers, see http://support.microsoft.com/kb/159199

For testing, I used regrtest -F -j6 test_import

Without patching it would fail consistently (albeit randomly).

I must also add that the proposed solution works well within the test suite as the access denied error can also occur when creating subsequent files, not just removing them.

This solution eliminates the need to wrap all creation calls with access denied handling, a huge plus IMHO.

I'm +1 on the approach in principle. I'm tentative about using ctypes
for this just because I don't believe we use it anywhere else. But at
the least I suggest applying the patch to see how Jeremy's buildbot
behaves. If there are wider objections to ctypes we could always work up
a C patch. (In 3.3+ we could use the new _winapi module).

My inclination is to say that using ctypes is a reasonable option for improving Windows buildbot stability in the near term, but we'd probably want to move this into _winapi long term.

Adding Antoine & MvL to get their opinion.

I wonder why it couldn't use os.listdir to find out whether the directory is empty, and os.stat to find out whether a specific file or directory still exists.

> On my machine I have programs that simply do not have an option to
> ignore any directories thus causing some grief during testing.

What are those programs exactly? A buildbot should ideally have a lean system install that does not interfere with the tests running.

> What are those programs exactly? A buildbot should ideally have a lean system install that does not interfere with the tests running.

My development machine has add'l programs, not the buildbot machine.
Sorry is there was any confusion.  I get the same occasional access
denied errors when running the tests locally.

> Tim Golden added the comment:
> I'm tentative about using ctypes this just because I don't believe we use it anywhere else.

ctypes is already used later in test_support so I figured it was fine
to use for this as well.

> I wonder why it couldn't use os.listdir to find out whether the directory is empty, and os.stat to find out whether a specific file or directory still exists.

It is possible to do the same thing with just os.listdir.  The use of
the Find*File functions was chosen to eliminate duplicating the wait
function for the two different cases.  Also,  it is just less resource
and time intensive to use the Find*File API directly (no need to build
an entire list when just testing if any entries exist).

> > I wonder why it couldn't use os.listdir to find out whether the directory is empty, and os.stat to find out whether a specific file or directory still exists.
> 
> It is possible to do the same thing with just os.listdir.  The use of
> the Find*File functions was chosen to eliminate duplicating the wait
> function for the two different cases.  Also,  it is just less resource
> and time intensive to use the Find*File API directly (no need to build
> an entire list when just testing if any entries exist).

But it's certainly much easier to maintain using regular Python APIs. We
would use ctypes if the functionality wasn't accessible from pure
Python, but since it is, we don't want to reinvent the wheel.

OK, here is another patch that uses just os.listdir

I fail to see the point of not using os.stat. IIUC, Windows will delete the file once the last handle is been closed. Since stat will close any handle it temporarily gets, it will not prolong the live of the file; the file will still go away when the last process has closed it.

Performance is not an issue at all here, since we are waiting for the deletion of the file anyway. So checking whether the file is in the directory listing is fine with me as well. Unless someone can demonstrate how os.stat can prevent removal of the file, I'd like to see the comment corrected, though.

I've updated the comment in the patch to reflect Martin's concern.

Martin is partially correct in that the handle opened in the stat() call will not prolong the pending status.  It is due to the fact that it does not open the handle with any sharing mode set, thus effectively blocking any other process from grabbing another handle to the file while the stat function has its handle open.

With the latest changes, is there anything left preventing the
inclusion of this patch?

Without some change, the Win64 buildbot is relatively irrelevant as it
is nearly always in a state of failure due to these errors.

> Without some change, the Win64 buildbot is relatively irrelevant as it
is nearly always in a state of failure due to these errors.

Not that some change isn't necessary, but what else are you running on your build slave? I ran a Windows 2008 R2 x64 slave for some time and it never had issues around file/directory removal. I only had to decommission it because the physical machine became unreliable.

> Not that some change isn't necessary, but what else are you running on your build slave? I ran a Windows 2008 R2 x64 slave for some time and it never had issues around file/directory removal. I only had to decommission it because the physical machine became unreliable.

The errors only started happening after upgrading the HD from a PATA
Ultra133 to an SATA3 SSD.  The super-fast HD is allowing for these
timing errors to show through.

Brian, Tim, do you think this should be committed?

The latest patch to test.support looks reasonable. Go for it.

Fine with me

New changeset fcad4566910b by Brian Curtin in branch '3.2':
Fix #15496. Add directory removal helpers to make Windows tests more reliable. Patch by Jeremy Kloth
http://hg.python.org/cpython/rev/fcad4566910b

New changeset c863dadc65eb by Brian Curtin in branch '2.7':
Fix #15496. Add directory removal helpers to make Windows tests more reliable. Patch by Jeremy Kloth
http://hg.python.org/cpython/rev/c863dadc65eb