classification
Title: subprocess: on Windows, unwanted file handles are inherited by child processes in a multithreaded application
Type: Stage:
Components: IO, Library (Lib), Windows Versions: Python 3.3, Python 3.4
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Bernt.Røskar.Brenna, astrand, gps, neologix, r.david.murray, sbt, tim.golden, vstinner
Priority: normal Keywords:

Created on 2013-11-13 21:02 by Bernt.Røskar.Brenna, last changed 2013-11-21 11:34 by vstinner.

Files
File name Uploaded Description Edit
repro.py Bernt.Røskar.Brenna, 2013-11-13 21:02
repro_improved.py Bernt.Røskar.Brenna, 2013-11-13 22:25
testcase3.py Bernt.Røskar.Brenna, 2013-11-14 13:09
testscript4.py Bernt.Røskar.Brenna, 2013-11-18 09:40
Messages (26)
msg202779 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-13 21:02
Running the following task using concurrent.futures.ThreadPoolExecutor works with max_workers == 1 and fails when max_workers > 1 :

def task():
    dirname = tempfile.mkdtemp()
    f_w =  open(os.path.join(dirname, "stdout.txt"), "w")
    f_r = open(os.path.join(dirname, "stdout.txt"), "r")
    e_w =  open(os.path.join(dirname, "stderr.txt"), "w")
    e_r = open(os.path.join(dirname, "stderr.txt"), "r")

    with subprocess.Popen("dir", shell=True, stdout=f_w, stderr=e_w) as p:
        pass

    f_w.close()
    f_r.close()
    e_w.close()
    e_r.close()

    shutil.rmtree(dirname)

The exception is raised by shutil.rmtree when max_workers > 1: "The process cannot access the file because it is being used by another process"

See also this Stack Overflow question about what seems to bee a similar problem: http://stackoverflow.com/questions/15966418/python-popen-on-windows-with-multithreading-cant-delete-stdout-stderr-logs  The discussion on SO indicates that this might be an XP problem only.

The attached file reproduces the problem on my Windows XP VM.
msg202780 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-13 21:06
Simpler task method that still reproduces the problem:

def task():
    dirname = tempfile.mkdtemp()
    f_w =  open(os.path.join(dirname, "stdout.txt"), "w")
    e_w =  open(os.path.join(dirname, "stderr.txt"), "w")

    with subprocess.Popen("dir", shell=True, stdout=f_w, stderr=e_w) as p:
        pass

    f_w.close()
    e_w.close()

    shutil.rmtree(dirname)
msg202783 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-13 21:38
It doesn't look like you are waiting for the subprocess to complete (and therefore close the files) before doing the rmtree.  What happens if you do that?
msg202785 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-13 21:39
Nevermind, I forgot that the context manager also does the wait.
msg202786 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-13 21:44
Most likely this is a case of a virus scanner or file indexer having the file open when rmtree runs.  See issue 7443, which was about solving this for our test suite.  My last message there asks about exposing the workaround in shutil, but no one appears to have taken me up on that suggestion yet.

Can you disable all virus scanners and indexers and try again?
msg202788 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-13 22:05
No indexing and no virus scanner running.

Note that it never fails if running in a single thread. IMO, this indicates that external processes interfering is not the problem.
msg202789 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-13 22:25
Here's an improved repro script.

I believe it demonstrates that it is the combination of subprocess.Popen and threading that causes the problem.

Here's the output from my Windows XP VM:

***

c:\...> c:\Python33\python.exe repro_improved.py
Windows-XP-5.1.2600-SP3
Concurrency: 2
Task kind: subprocess_redirfile
3 errors of 10

Concurrency: 1
Task kind: subprocess_redirfile
0 errors of 10

Concurrency: 2
Task kind: subprocess_devnull
5 errors of 10

Concurrency: 1
Task kind: subprocess_devnull
0 errors of 10

Concurrency: 2
Task kind: nosubprocess
0 errors of 10

Concurrency: 1
Task kind: nosubprocess
0 errors of 10

***

Note that:
- even when subprocess redirects to DEVNULL there are errors
- when no subprocess.Popen is executed, no errors occur (the file is created as normal, but is not used by subprocess.Popen)
msg202839 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-14 13:09
Another script, another test case.

Four different tasks are run:

subprocess_redirfile: Popen(stdout=file)
subprocess_devnull: Popen(stdout=DEVNULL)
subprocess_noredirect: Popen()
nosubprocess: No Popen() call

Judging from the output it looks as if it is the redirection that triggers this behavior.

Here's the output from my Win XP computer:

Platform: Windows-XP-5.1.2600-SP3

task_type                 #threads   result    
subprocess_redirfile      2          4 errors  
subprocess_redirfile      1          OK        
subprocess_devnull        2          5 errors  
subprocess_devnull        1          OK        
subprocess_noredirect     2          OK        
subprocess_noredirect     1          OK        
nosubprocess              2          OK        
nosubprocess              1          OK
msg202843 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-14 14:04
Similar result on Windows Server 2008:

Platform: Windows-2008ServerR2-6.1.7600

task_type                 #threads   result
subprocess_redirfile      2          9 errors
subprocess_redirfile      1          OK
subprocess_devnull        2          9 errors
subprocess_devnull        1          OK
subprocess_noredirect     2          OK
subprocess_noredirect     1          OK
nosubprocess              2          OK
nosubprocess              1          OK
msg202852 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-14 15:39
Unfortunately I currently lack a windows environment on which to test this.  I've added some people to nosy who might be able to help out with this.
msg202854 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-11-14 15:59
I think it's simply due to file descriptor inheritance (files being
inherited by other subprocess instance): since Windows can't remove open
files, kaboom. It doesn't have anything to do with threads.
msg202859 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-14 16:49
That was my initial thought, too, but the subprocess context manager waits for the subprocess to end, so those file descriptors should be closed by the time the deletion attempt happens, shouldn't they?
msg202872 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-11-14 18:20
Note that on Windows if you redirect the standard streams then *all* inheritable handles are inherited by the child process.

Presumably the handle for f_w file object (and/or a duplicate of it) created in one thread is accidentally "leaked" to the other child process.  This means that shutil.rmtree() cannot succeed until *both* child processes have exited.

PEP 446 might fix this, although there will still be a race condition.
msg202876 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-14 19:24
@neologix: How can it not have anything to do with threads? 

- It always works with max_workers == 1
- When max_workers == 2, shutil.rmtree sometimes fails
msg202877 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-11-14 19:43
> Bernt Røskar Brenna added the comment:
>
> @neologix: How can it not have anything to do with threads?
>
> - It always works with max_workers == 1
> - When max_workers == 2, shutil.rmtree sometimes fails

It has nothing to do with threads.
You could reproduce it by opening your files, spawning two child
processes, wait until the first one returns, and then try to remove
the files used by the first subprocess.
msg202894 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-14 21:23
@neologix: But how do you explain this:

subprocess_devnull        2          9 errors
subprocess_devnull        1          OK

subprocess_devnull creates a file, then starts a subprocess (that redirects to DEVNULL, does not use the file), then tries to remove the directory containing the file. When running in parallel, it fails.

subprocess_noredirect     2          OK
subprocess_noredirect     1          OK

subprocess_noredirect creates a file, then starts a subprocess (that does not use the file), then tries to remove the directory containing the file. When running in parallel, it does not fail.
msg202896 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-14 21:40
neologix noted that *when redirection is used* the way that *all* windows file handles are inherited changes.  But that's about the end of *my* understanding of the issue :)
msg202985 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2013-11-16 00:11
> R. David Murray added the comment:
>
> neologix noted that *when redirection is used* the way that *all* windows file handles are inherited changes.

That's true (but that was from Richard actually).
msg202987 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-16 00:14
> I think it's simply due to file descriptor inheritance

File handles are now non-inheritable by default in Python 3.4 (PEP 446). Does it change anything?
msg203266 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-18 09:30
I just tested on 3.4a0.

Observed the following changes: 

- subprocess_devnull now NEVER fails.
- subprocess_redirfile does not fail as often as before, but still fails.

I changed the number of tasks to 20 and increased max_workers to 5 to get subprocess_redirfile to fail at least one of twenty times every time I invoked the test script.

A typical result on 3.4 looks like this:

Platform: Windows-XP-5.1.2600-SP3

task_type                 #threads   result
subprocess_redirfile      5          3 errors
subprocess_redirfile      1          OK
subprocess_devnull        5          OK
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK
msg203269 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-18 09:40
Increasing max_workers to 5 and running 20 tasks highlighs the 3.3-3.4 difference (code in attached file testscript4.py):

Version: 3.4.0a4 (v3.4.0a4:e245b0d7209b, Oct 20 2013, 19:23:45) [MSC v.1600 32 b
it (Intel)]
Platform: Windows-XP-5.1.2600-SP3
Tasks: 20

task_type                 #threads   result
subprocess_redirfile      5          4 errors (of 20)
subprocess_redirfile      1          OK
subprocess_devnull        5          OK
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK



Version: 3.3.2 (v3.3.2:d047928ae3f6, May 16 2013, 00:03:43) [MSC v.1600 32 bit (
Intel)]
Platform: Windows-XP-5.1.2600-SP3
Tasks: 20

task_type                 #threads   result
subprocess_redirfile      5          18 errors (of 20)
subprocess_redirfile      1          OK
subprocess_devnull        5          19 errors (of 20)
subprocess_devnull        1          OK
subprocess_noredirect     5          OK
subprocess_noredirect     1          OK
nosubprocess              5          OK
nosubprocess              1          OK
msg203276 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-18 10:48
Ok, this issue is the corner case of the PEP 446:
http://www.python.org/dev/peps/pep-0446/#only-inherit-some-handles-on-windows

The PEP explicitly does nothing for this case. It can change in the future.

Until the point is fixed, you have to use a lock around the code spawning new processes to avoid that two threads spawn two processes and inherit unexpected files.

Example: Thread 1 creates file 1, thread 2 creates file 2, child process inherits files 1 and 2, instead of just file 1.

Richard proposed to use a trampoline process, the parent process would it the handles to inherit. Since Windows Vista, the trampoline process is no more needed.
msg203280 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-18 11:28
I tested, and locking around the subprocess.Popen call indeed works on Python 3.4. 

Thanks!

Do you have any tips on how to accomplish the same thing on Python 3.3 (locking around Popen did not make any difference on 3.3)?
msg203282 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-18 11:43
Never mind, I figured it:

On Python 3.3, the combination of locking around Popen and opening the file that I redirect to using the code below works (code from scons):

def open_noinherit(*args, **kwargs):
    fp = open(*args, **kwargs)
    win32api.SetHandleInformation(msvcrt.get_osfhandle(fp.fileno()),
                                  win32con.HANDLE_FLAG_INHERIT,
                                  0)
    return fp
msg203288 - (view) Author: Bernt Røskar Brenna (Bernt.Røskar.Brenna) * Date: 2013-11-18 12:42
And here's a function that does not require pywin32:

def open_noinherit_ctypes(*args, **kwargs):
    HANDLE_FLAG_INHERIT = 1

    import msvcrt
    from ctypes import windll, WinError
    fp = open(*args, **kwargs)
    if not windll.kernel32.SetHandleInformation(msvcrt.get_osfhandle(fp.fileno()), HANDLE_FLAG_INHERIT, 0):
        raise WinError()
    return fp
msg203596 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-21 09:34
In Python 3.3, open_noinherit_ctypes() can be written:

def opener_noinherit(filename, flags):
    return os.open(filename, flags | os.O_NOINHERIT)
f = open(filename, opener=opener_noinherit)


Example on Linux with O_CLOEXEC:

$ python3
Python 3.3.0 (default, Sep 29 2012, 22:07:38) 
[GCC 4.7.2 20120921 (Red Hat 4.7.2-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> def opener_noinherit(filename, flags):
...     return os.open(filename, flags | os.O_CLOEXEC)
... 
>>> f=open("/etc/issue", opener=opener_noinherit)
>>> import fcntl
>>> fcntl.fcntl(f.fileno(), fcntl.F_GETFD) & fcntl.FD_CLOEXEC
1
History
Date User Action Args
2013-11-21 11:34:14vstinnersettitle: subprocess: on Windows, unwanted file handles are inherit by child processes in a multithreaded application -> subprocess: on Windows, unwanted file handles are inherited by child processes in a multithreaded application
2013-11-21 09:35:53vstinnersettitle: subprocess.Popen with multiple threads: Redirected stdout/stderr files still open after process close -> subprocess: on Windows, unwanted file handles are inherit by child processes in a multithreaded application
2013-11-21 09:34:40vstinnersetmessages: + msg203596
2013-11-18 12:42:32Bernt.Røskar.Brennasetmessages: + msg203288
2013-11-18 11:43:36Bernt.Røskar.Brennasetmessages: + msg203282
2013-11-18 11:28:15Bernt.Røskar.Brennasetmessages: + msg203280
2013-11-18 10:48:59vstinnersetmessages: + msg203276
2013-11-18 10:42:34Bernt.Røskar.Brennasetcomponents: + IO
2013-11-18 09:54:31Bernt.Røskar.Brennasetversions: + Python 3.4
2013-11-18 09:40:50Bernt.Røskar.Brennasetfiles: + testscript4.py

messages: + msg203269
2013-11-18 09:30:54Bernt.Røskar.Brennasetmessages: + msg203266
2013-11-16 00:14:03vstinnersetnosy: + vstinner
messages: + msg202987
2013-11-16 00:11:50neologixsetmessages: + msg202985
2013-11-14 21:40:29r.david.murraysetmessages: + msg202896
2013-11-14 21:23:01Bernt.Røskar.Brennasetmessages: + msg202894
2013-11-14 19:43:41neologixsetmessages: + msg202877
2013-11-14 19:42:01Bernt.Røskar.Brennasetnosy: + astrand
2013-11-14 19:24:29Bernt.Røskar.Brennasetmessages: + msg202876
2013-11-14 18:20:05sbtsetnosy: + sbt
messages: + msg202872
2013-11-14 16:49:25r.david.murraysetmessages: + msg202859
2013-11-14 15:59:03neologixsetnosy: + neologix
messages: + msg202854
2013-11-14 15:39:24r.david.murraysetnosy: + gps, tim.golden
messages: + msg202852
2013-11-14 14:04:15Bernt.Røskar.Brennasetmessages: + msg202843
2013-11-14 13:09:53Bernt.Røskar.Brennasetfiles: + testcase3.py

messages: + msg202839
2013-11-13 22:25:52Bernt.Røskar.Brennasetfiles: + repro_improved.py

messages: + msg202789
2013-11-13 22:05:38Bernt.Røskar.Brennasetmessages: + msg202788
2013-11-13 21:44:36r.david.murraysetmessages: + msg202786
2013-11-13 21:39:27r.david.murraysetmessages: + msg202785
2013-11-13 21:38:20r.david.murraysetnosy: + r.david.murray
messages: + msg202783
2013-11-13 21:06:15Bernt.Røskar.Brennasetmessages: + msg202780
2013-11-13 21:02:53Bernt.Røskar.Brennacreate