classification
Title: subprocess.py throw "The handle is invalid" error on duplicating the STD_INPUT_HANDLE
Type: behavior Stage: resolved
Components: Windows Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder: subprocess failing in GUI applications on Windows
View: 3905
Assigned To: Nosy List: BreamoreBoy, gregory.p.smith, hercs, kartlee05, tim.golden
Priority: normal Keywords:

Created on 2012-11-12 02:49 by kartlee05, last changed 2015-12-05 19:44 by eryksun. This issue is now closed.

Files
File name Uploaded Description Edit
subprocess.zip kartlee05, 2012-11-12 02:49 Test files to reproduce the issue and patch
Messages (9)
msg175416 - (view) Author: Karthk Rajagopalan (kartlee05) Date: 2012-11-12 02:49
Please download subprocess.zip and extract the archive. Run - 

perl test.pl

you will notice 'invalid handle' error on duplicating STD_INPUT_HANDLE in python. This is observed in Windows XP SP3. Vista SP2 and higher platforms doesn't show this failure when I checked.

"test.pl" spawn a python process "test.py" using CREATE_NEW_PROCESS_GROUP flag. Before it does, it close STDIN and create a listening socket. Perl under the hood, convert the socket handle to file descriptor using open_osfhandle(..) call. When it does this, the C runtime set the standard input handle to this descriptor  for console app. [ See msvcr* osfinfo.c code - SetStdHandle( STD_INPUT_HANDLE, (HANDLE)value )  ]. As a result, when you get the STD_INPUT_HANDLE, you will get the new one. Duplicating this handle, hence throw the invalid handle error in Windows XP SP3. 

When I looked at MSDN for DuplicateHandle(..), it is mentioned not to duplicate socket handle using this call. So I suspect something going wrong in XP platform. To fix this issue, I have come up with a patch - python-2143_patch. Please take a look and let me know your comments.

-Karthik
msg175522 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2012-11-13 20:29
I'm sorry, but I genuinely can't see what you're trying to say here. If you believe that there's a bug in Python's standard library, can you show a reproducible *Python* testcase for it, please (using the pywin32 modules if that helps) and state clearly what you think is wrong.
msg175526 - (view) Author: Karthk Rajagopalan (kartlee05) Date: 2012-11-13 20:57
Hi Tim,

Thanks for your reply.

I added test case using perl and python since it was easy to reproduce using perl socket module and show the issue happening with python's subprocess.py. There is definitely an action required in subprocess.py to catch 'invalid handle error' and my attached patch handle this situation.

I can come up with a test case like perl does to create a listening socket and map the socket handle to a file descriptor. You will see this issue *only* when the open_osfhandle(..) is used to map the handle to a file descriptor.

-Karthik
msg175571 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2012-11-14 15:20
On 13/11/2012 20:57, Karthk Rajagopalan wrote:
> I added test case using perl and python since it was easy to
> reproduce using perl socket module and show the issue happening with
> python's subprocess.py. There is definitely an action required in
> subprocess.py to catch 'invalid handle error' and my attached patch
> handle this situation.

So, because subprocess.call includes a call to DuplicateHandle(stdin),
that call will fail if stdin is currently a handle to a socket.
Is that right?

Your patch fails to apply cleanly to the hg tip, because Gregory P.
Smith has been making changes in this very piece of code over the last
few days. I'll add him as interested to the call as someone who can make
a call as to the suitability of your patch.

> I can come up with a test case like perl does to create a listening
> socket and map the socket handle to a file descriptor. You will see
> this issue *only* when the open_osfhandle(..) is used to map the
> handle to a file descriptor.

If this is to go ahead, we will need a test case which can run as part
of the Python stdlib tests, and perl is not part of that stdlib!
msg175572 - (view) Author: Karthk Rajagopalan (kartlee05) Date: 2012-11-14 15:34
Hi Tim, 

Thanks for your reply.

Yes, DuplicateHandle(..) seem to fail if the handle is to a socket under XP SP3. 

Can you point me to the guidelines about submitting patch so I can merge my change in main branch and upload it?

We build python from source and doesn't use pywin32 as of now. So I downloaded them yesterday to come up with a test case in python which you can use in your test bed.

-Karthik
msg175573 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2012-11-14 15:37
Start here: http://docs.python.org/devguide/

In particular: http://docs.python.org/devguide/patch.html
msg175587 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2012-11-14 20:22
Sorry I can't help on the windows side of things.  I don't have access to any windows systems.
msg175637 - (view) Author: Tim Golden (tim.golden) * (Python committer) Date: 2012-11-15 20:31
Karthk, if you can run up an up-to-date patch and a test I'm willing to review and commit it, otherwise this one will lie quiet.
msg222956 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2014-07-13 17:30
Is XP under Python support now?  I'm sure I've read that it isn't but can't find a reference.
History
Date User Action Args
2015-12-05 19:44:48eryksunsetstatus: open -> closed
superseder: subprocess failing in GUI applications on Windows
resolution: duplicate
stage: resolved
2014-07-13 17:30:49BreamoreBoysetnosy: + BreamoreBoy
messages: + msg222956
2012-11-15 20:31:01tim.goldensetmessages: + msg175637
2012-11-14 20:22:01gregory.p.smithsetnosy: gregory.p.smith, tim.golden, kartlee05, hercs
messages: + msg175587
2012-11-14 15:37:38tim.goldensetmessages: + msg175573
2012-11-14 15:34:35kartlee05setmessages: + msg175572
2012-11-14 15:20:32tim.goldensetnosy: + gregory.p.smith
2012-11-14 15:20:01tim.goldensetmessages: + msg175571
2012-11-13 20:57:50kartlee05setmessages: + msg175526
2012-11-13 20:29:24tim.goldensetnosy: + tim.golden
messages: + msg175522
2012-11-12 19:43:26hercssetnosy: + hercs
2012-11-12 02:49:27kartlee05create