I'm getting a 'The handle is invalid' error when using subprocess.Popen
in Python 2.5 (and 2.6). If any of the stdio handles are somehow invalid
or not real io handles (fd is not 0, 1 or 2), and you are not telling
subprocess to PIPE all of the handles, then subprocess throws the
following exception.

The easiest way to reproduce this is using run PythonWin from a Windows
Command Prompt:
  C:\Python\Lib\site-packages\pythonwin\Pythonwin.exe
and then use the following 2 subprocess code lines below:

import subprocess
p = subprocess.Popen(["python", "-c", "print 32"], stdin=None,
stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Traceback (most recent call last):
  File "<interactive input>", line 1, in <module>
  File "C:\Python26\lib\subprocess.py", line 588, in init_
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\Python26\lib\subprocess.py", line 703, in _get_handles
    p2cread = self._make_inheritable(p2cread)
  File "C:\Python26\lib\subprocess.py", line 748, in _make_inheritable
    DUPLICATE_SAME_ACCESS)
WindowsError: [Error 6] The handle is invalid


Specifying PIPE for all subprocess.Popen handles is the only way I've
found to get around this problem:

p = subprocess.Popen(["python", "-c", "print 32"],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.stdin.close()
p.communicate()
('32\r\n', '')

This seems to be somewhat related to issue1124861:
http://bugs.python.org/issue1124861

The failure is in the DuplicateHandle call that subprocess makes on, in
this case, the stdin handle (as returned by
`GetStdHandle(STD_INPUT_HANDLE)`) call earlier.

Two cases here:

1. When this is run in a subsystem:windows process (like PythonWin or
IDLE) that is launched from Windows Explorer (e.g. from a Start Menu
shortcut or Desktop shortcut) then `GetStdHandle(STD_INPUT_HANDLE)`
returns None. 

[http://msdn.microsoft.com/en-us/library/ms683231(VS.85).aspx]
> If an application does not have associated standard handles,
> such as a service running on an interactive desktop, and has
> not redirected them, the return value is NULL.

In this case you *don't* get the error that Todd described, because the
code path taken in subprocess.py then use CreatePipe for the `p2cread`
variable on which `DuplicateHandle` is called.

2. However, when the subsystem:windows process is launched from the
cmd.exe shell the `GetStdHandle` call returns a value -- in Todd's and
my testing, the value 3.

The code path in subprocess.py then calls `DuplicateHandle` on this in
`Popen._make_inheritable`. This fails with traceback Todd posted.

My *guess* at what the problem is stems from this comment in the MSDN
docs on console handles:

[http://msdn.microsoft.com/en-us/library/ms682075(VS.85).aspx]
> A process can use the DuplicateHandle function to create a 
> duplicate console handle that has different access or 
> inheritability from the original handle. Note, however,
> that a process can create a duplicate console handle only
> for its own use. This differs from other handle types (such
> as file, pipe, or mutex objects), for which DuplicateHandle
> can create a duplicate that is valid for a different process.

My guess is that the stdin handle (3) is inherited from the shell
(cmd.exe) and attempting to `DuplicateHandle` on it violates the clause
that you can on dupe a console handle of your own. I'm not sure of that
though.

Anyone else have more light to shed on this?

If this is the case I'm not sure what a possible or good solution could
be for subprocess.

I do not really know if this is related to this, but I suspect yes.
On my w2k+sp4 box, swiss french, Python 3.0rc1, IDLE does not start or
more precisely it starts by opening the following message box:

Subprocess Startup Error
---------------------------
IDLE's subprocess didn't make connection.  Either IDLE can't start a
subprocess or personal firewall software is blocking the connection.

It is certainly neither a firewall issue, nor a tkinter issue (tkinter 
applications are working fine.)

No problem with the 3.0b2 realease, 3.0b3: not tested.

jmfauth: please try to run idle from a command prompt:

> cd <path_to_python3.0rc1>
> python Lib/idlelib/idle.py

Do you see interesting output there?

Microsoft Windows 2000 [Version 5.00.2195]
(C) Copyright 1985-2000 Microsoft Corp.

C:\>cd python30

C:\Python30>python Lib/idlelib/idle.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python30\lib\idlelib\run.py", line 76, in main
    sockthread.set_daemon(True)
AttributeError: 'Thread' object has no attribute 'set_daemon'

C:\Python30>

Sorry if I can help here, things like socket and subprocess are not my
cup of tea.

Benjamin, I think you're responsible.

There are also instances of set_daemon left in socketserver and
multiprocessing/dummy. How is that possible?

The idle problem has already been fixed, and I got the socket server one
in r66520.

Just for information and from an end user perspective.

I have tried to replace the socketserver.py from the original 3.0rc1
distribution by the socketserver.py as proposed by Benjamin Peterson
(r66520).

Script difference (line 568):

        if self.daemon_threads:
            t.daemon = True
        t.start()

and 

        if self.daemon_threads:
            t.set_daemon(True)
        t.start()

Unfortunately, no luck, I'm still getting exactly the same error
messages, the msg box and the raised AttributeError:

AttributeError: 'Thread' object has no attribute 'set_daemon'

Jean-Michel, you'll need to apply the idlelib/run.py patch; replace

    sockthread.set_daemon(True)

with

    sockthread.daemon = True

around line 75.

Uh, the Idle bug reported by Jean-Michel is a completely different bug
(please see the first three messages of this report).

Please re-open this bug, as the subprocess issue I have reported is
still outstanding.

> Georg Brandl

Thanks, patch applied on w2k+sp4 box, swiss french, Python 3.0rc1.
IDLE is working fine.

Georg Brandl thank you, i was having the same problem, but now is fixed

I just re-tested this in cpython trunk at changeset and the issue does not appear to be reproducible.

I first launched IDLE by running "python lib\idlelib\idle.py".  Then I entered the following:

Python 3.3a0 (default, Apr  2 2011, 21:55:40) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import sys
>>> sys.getwindowsversion()
sys.getwindowsversion(major=5, minor=1, build=2600, platform=2, service_pack='Service Pack 3')
>>> import subprocess
>>> PIPE = subprocess.PIPE
>>> p = subprocess.Popen(["python", "-c", "print(32)"], stdin=None, stdout=PIPE, stderr=PIPE)
>>> p
<subprocess.Popen object at 0x00F85610>
>>> p.returncode
>>>

Popen() did not raise any exceptions.  I also tried in Python 2.7 and the Popen called succeeded there as well.  So my inability to reproduce the issue does not necessarily mean that the issue is fixed, but rather that it is likely dependent on the version of Windows on which Python is running.

That being said, the linked issue issue1124861 logs what appears to be the same issue and it was fixed in Python 2.6.  So maybe I'm just not old enough to have encountered this issue :)  Anyways, this issue probably can be closed as either a duplicate or fixed incidentally.

I still see this problem (in Python 2.7.1 and Python 3.1.2).

When testing using idle, you'll need to launch using "pythonw.exe", i.e.:

pythonw.exe lib\idlelib\idle.py

Ahh okay.  I've reproduced it in trunk at changeset 053bc5ca199b.

As suggested, I ran: PCBuild\pythonw.exe lib\idlelib\idle.py

Python 3.3a0 (default, Apr  2 2011, 21:55:40) [MSC v.1500 32 bit (Intel)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import subprocess
>>> subprocess.Popen(["python", "-c", "print(32)"], stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Traceback (most recent call last):
  File "<pyshell#1>", line 1, in <module>
    subprocess.Popen(["python", "-c", "print(32)"], stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "C:\dev\cpython\cpython\lib\subprocess.py", line 732, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\dev\cpython\cpython\lib\subprocess.py", line 903, in _get_handles
    p2cread = self._make_inheritable(p2cread)
  File "C:\dev\cpython\cpython\lib\subprocess.py", line 946, in _make_inheritable
    _subprocess.DUPLICATE_SAME_ACCESS)
WindowsError: [Error 6] The handle is invalid
>>> 

So, the issue is definitely not fixed.

In 3.2.2, starting IDLE from a Command Prompt, I get the same error as Denver. Starting IDLE 'normally', from the icon, there is no error. So even though Command Prompt detaches pythonw in some way and gives me a new prompt to do something else, there is something different about the pythonw process that trips up subprocess.

Changing ["python", "-c", "print(32)"] to ["notepad"] gives the same result. Running in the interactive interpreted eliminated the error. So does changing stdin=None to stdin=subprocess.PIPE eliminates the error and brings up Notepad, which seems to operate normally. (I am guessing that it ignores stdin, etc.).

I see that that the latter was known in the original post and plausibly diagnosed by Trent Mick, but never fixed because it was not obvious how to do so.

Can the WindowsError be caught in subproccess, and if it is this error (6), follow the alternate patch that is used for NULL returns? Could that cause worse problems that the current situation?

If this is unfixable due to Windows behavior, perhaps we should say something in the doc.