Running the following task using concurrent.futures.ThreadPoolExecutor works with max_workers == 1 and fails when max_workers > 1 :

def task():
    dirname = tempfile.mkdtemp()
    f_w =  open(os.path.join(dirname, "stdout.txt"), "w")
    f_r = open(os.path.join(dirname, "stdout.txt"), "r")
    e_w =  open(os.path.join(dirname, "stderr.txt"), "w")
    e_r = open(os.path.join(dirname, "stderr.txt"), "r")

    with subprocess.Popen("dir", shell=True, stdout=f_w, stderr=e_w) as p:
        pass

    f_w.close()
    f_r.close()
    e_w.close()
    e_r.close()

    shutil.rmtree(dirname)

The exception is raised by shutil.rmtree when max_workers > 1: "The process cannot access the file because it is being used by another process"

See also this Stack Overflow question about what seems to bee a similar problem: http://stackoverflow.com/questions/15966418/python-popen-on-windows-with-multithreading-cant-delete-stdout-stderr-logs  The discussion on SO indicates that this might be an XP problem only.

The attached file reproduces the problem on my Windows XP VM.

Simpler task method that still reproduces the problem:

def task():
    dirname = tempfile.mkdtemp()
    f_w =  open(os.path.join(dirname, "stdout.txt"), "w")
    e_w =  open(os.path.join(dirname, "stderr.txt"), "w")

    with subprocess.Popen("dir", shell=True, stdout=f_w, stderr=e_w) as p:
        pass

    f_w.close()
    e_w.close()

    shutil.rmtree(dirname)

It doesn't look like you are waiting for the subprocess to complete (and therefore close the files) before doing the rmtree.  What happens if you do that?

Nevermind, I forgot that the context manager also does the wait.

Most likely this is a case of a virus scanner or file indexer having the file open when rmtree runs.  See issue 7443, which was about solving this for our test suite.  My last message there asks about exposing the workaround in shutil, but no one appears to have taken me up on that suggestion yet.

Can you disable all virus scanners and indexers and try again?

No indexing and no virus scanner running.

Note that it never fails if running in a single thread. IMO, this indicates that external processes interfering is not the problem.

Here's an improved repro script.

I believe it demonstrates that it is the combination of subprocess.Popen and threading that causes the problem.

Here's the output from my Windows XP VM:

***

c:\...> c:\Python33\python.exe repro_improved.py
Windows-XP-5.1.2600-SP3
Concurrency: 2
Task kind: subprocess_redirfile
3 errors of 10

Concurrency: 1
Task kind: subprocess_redirfile
0 errors of 10

Concurrency: 2
Task kind: subprocess_devnull
5 errors of 10

Concurrency: 1
Task kind: subprocess_devnull
0 errors of 10

Concurrency: 2
Task kind: nosubprocess
0 errors of 10

Concurrency: 1
Task kind: nosubprocess
0 errors of 10

***

Note that:
- even when subprocess redirects to DEVNULL there are errors
- when no subprocess.Popen is executed, no errors occur (the file is created as normal, but is not used by subprocess.Popen)

Another script, another test case.

Four different tasks are run:

subprocess_redirfile: Popen(stdout=file)
subprocess_devnull: Popen(stdout=DEVNULL)
subprocess_noredirect: Popen()
nosubprocess: No Popen() call

Judging from the output it looks as if it is the redirection that triggers this behavior.

Here's the output from my Win XP computer:

Platform: Windows-XP-5.1.2600-SP3

task_type                 #threads   result    
subprocess_redirfile      2          4 errors  
subprocess_redirfile      1          OK        
subprocess_devnull        2          5 errors  
subprocess_devnull        1          OK        
subprocess_noredirect     2          OK        
subprocess_noredirect     1          OK        
nosubprocess              2          OK        
nosubprocess              1          OK

Similar result on Windows Server 2008:

Platform: Windows-2008ServerR2-6.1.7600

task_type                 #threads   result
subprocess_redirfile      2          9 errors
subprocess_redirfile      1          OK
subprocess_devnull        2          9 errors
subprocess_devnull        1          OK
subprocess_noredirect     2          OK
subprocess_noredirect     1          OK
nosubprocess              2          OK
nosubprocess              1          OK

Unfortunately I currently lack a windows environment on which to test this.  I've added some people to nosy who might be able to help out with this.

I think it's simply due to file descriptor inheritance (files being
inherited by other subprocess instance): since Windows can't remove open
files, kaboom. It doesn't have anything to do with threads.

That was my initial thought, too, but the subprocess context manager waits for the subprocess to end, so those file descriptors should be closed by the time the deletion attempt happens, shouldn't they?

Note that on Windows if you redirect the standard streams then *all* inheritable handles are inherited by the child process.

Presumably the handle for f_w file object (and/or a duplicate of it) created in one thread is accidentally "leaked" to the other child process.  This means that shutil.rmtree() cannot succeed until *both* child processes have exited.

PEP 446 might fix this, although there will still be a race condition.

@neologix: How can it not have anything to do with threads? 

- It always works with max_workers == 1
- When max_workers == 2, shutil.rmtree sometimes fails

> Bernt Røskar Brenna added the comment:
>
> @neologix: How can it not have anything to do with threads?
>
> - It always works with max_workers == 1
> - When max_workers == 2, shutil.rmtree sometimes fails

It has nothing to do with threads.
You could reproduce it by opening your files, spawning two child
processes, wait until the first one returns, and then try to remove
the files used by the first subprocess.

@neologix: But how do you explain this:

subprocess_devnull        2          9 errors
subprocess_devnull        1          OK

subprocess_devnull creates a file, then starts a subprocess (that redirects to DEVNULL, does not use the file), then tries to remove the directory containing the file. When running in parallel, it fails.

subprocess_noredirect     2          OK
subprocess_noredirect     1          OK

subprocess_noredirect creates a file, then starts a subprocess (that does not use the file), then tries to remove the directory containing the file. When running in parallel, it does not fail.

neologix noted that *when redirection is used* the way that *all* windows file handles are inherited changes.  But that's about the end of *my* understanding of the issue :)

> R. David Murray added the comment:
>
> neologix noted that *when redirection is used* the way that *all* windows file handles are inherited changes.

That's true (but that was from Richard actually).

> I think it's simply due to file descriptor inheritance

File handles are now non-inheritable by default in Python 3.4 (PEP 446). Does it change anything?

I just tested on 3.4a0.

Observed the following changes: 

- subprocess_devnull now NEVER fails.
- subprocess_redirfile does not fail as often as before, but still fails.

I changed the number of tasks to 20 and increased max_workers to 5 to get subprocess_redirfile to fail at least one of twenty times every time I invoked the test script.

A typical result on 3.4 looks like this:

Platform: Windows-XP-5.1.2600-SP3

task_type                 #threads   result
subprocess_redirfile      5          3 errors
subprocess_redirfile      1          OK
subprocess_devnull        5          OK
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK

Increasing max_workers to 5 and running 20 tasks highlighs the 3.3-3.4 difference (code in attached file testscript4.py):

Version: 3.4.0a4 (v3.4.0a4:e245b0d7209b, Oct 20 2013, 19:23:45) [MSC v.1600 32 b
it (Intel)]
Platform: Windows-XP-5.1.2600-SP3
Tasks: 20

task_type                 #threads   result
subprocess_redirfile      5          4 errors (of 20)
subprocess_redirfile      1          OK
subprocess_devnull        5          OK
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK



Version: 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (
Intel)]
Platform: Windows-XP-5.1.2600-SP3
Tasks: 20

task_type                 #threads   result
subprocess_redirfile      5          18 errors (of 20)
subprocess_redirfile      1          OK
subprocess_devnull        5          19 errors (of 20)
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK

Ok, this issue is the corner case of the PEP 446:
http://www.python.org/dev/peps/pep-0446/#only-inherit-some-handles-on-windows

The PEP explicitly does nothing for this case. It can change in the future.

Until the point is fixed, you have to use a lock around the code spawning new processes to avoid that two threads spawn two processes and inherit unexpected files.

Example: Thread 1 creates file 1, thread 2 creates file 2, child process inherits files 1 and 2, instead of just file 1.

Richard proposed to use a trampoline process, the parent process would it the handles to inherit. Since Windows Vista, the trampoline process is no more needed.

I tested, and locking around the subprocess.Popen call indeed works on Python 3.4. 

Thanks!

Do you have any tips on how to accomplish the same thing on Python 3.3 (locking around Popen did not make any difference on 3.3)?

Never mind, I figured it:

On Python 3.3, the combination of locking around Popen and opening the file that I redirect to using the code below works (code from scons):

def open_noinherit(*args, **kwargs):
    fp = open(*args, **kwargs)
    win32api.SetHandleInformation(msvcrt.get_osfhandle(fp.fileno()),
                                  win32con.HANDLE_FLAG_INHERIT,
                                  0)
    return fp

And here's a function that does not require pywin32:

def open_noinherit_ctypes(*args, **kwargs):
    HANDLE_FLAG_INHERIT = 1

    import msvcrt
    from ctypes import windll, WinError
    fp = open(*args, **kwargs)
    if not windll.kernel32.SetHandleInformation(msvcrt.get_osfhandle(fp.fileno()), HANDLE_FLAG_INHERIT, 0):
        raise WinError()
    return fp

In Python 3.3, open_noinherit_ctypes() can be written:

def opener_noinherit(filename, flags):
    return os.open(filename, flags | os.O_NOINHERIT)
f = open(filename, opener=opener_noinherit)


Example on Linux with O_CLOEXEC:

$ python3
Python 3.3.0 (default, Sep 29 2012, 22:07:38) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> def opener_noinherit(filename, flags):
...     return os.open(filename, flags | os.O_CLOEXEC)
... 
>>> f=open("/etc/issue", opener=opener_noinherit)
>>> import fcntl
>>> fcntl.fcntl(f.fileno(), fcntl.F_GETFD) & fcntl.FD_CLOEXEC
1