classification
Title: popen2 on Windows does not check _fdopen return value
Type: crash Stage: resolved
Components: Interpreter Core, Windows Versions: Python 2.7, Python 2.6
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, Ramchandra Apte, christian.heimes, frankty, rupole, terry.reedy
Priority: high Keywords: easy

Created on 2004-01-28 20:28 by frankty, last changed 2013-03-25 20:51 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
bug.zip frankty, 2004-01-28 20:28 zip with test case and hacks
Messages (7)
msg19840 - (view) Author: Frank Terhaar-Yonkers (frankty) Date: 2004-01-28 20:28
WinPython 2.3.3 crashes when using popen2 to spawn
*lots* of children > 100.

Running the test case on either W2k or Wxp causes
Python 2.3.3 to crash.  Looking at the drwatson file,
it appears to be a race condition where the child is in
the process of crashing, while the popen2 code in
posixmodule.c is attempting to PyFile_SetBufSize() on
one of the pipes.  Evil juju in ntdll.dll ..

I've no idea what system resource is being exhausted
(heap memory?) causing the crash.

If you look at the hacks in posixmodule.c, the Python
crash can be eliminated by simply commenting out the
calls to PyFile_SetBufSize.  There is still the problem
of resource exhaustion.  To gain more child processes I
included a hack such that with the setting of an
env-var (NOPYCMD) popen2 will NOT use cmd.exe /c to run
a child, but simply run the child command as passed
(yeah, I know that is has to be an absolute path or in
the CWD then ..).

I'm sure there is a better way to eliminate the cmd.exe
/c - like maybe a new method: popen2.popenXabspath ..

In any case, it appears we have an NT kernel bug
combined with an inefficient way of doing popen2 that
is making a mess.  I started doing some exploration of
heap_chunk_size to attack to resource issue but haven't
gotten very far.

As for *why* I'm doing this, it's because we're using
Pyton as a system simulator tool to emulate a large
pile of PCs.  The actual work *must* be done by
external .exe code that cannot be changed.

The attached zip contains a test case, the posixmodule
hack, drwatson file and a sample program written in
VisC++ which does basically the same as the Python test
case.

cheers,

- Frank

msg19841 - (view) Author: Roger Upole (rupole) Date: 2007-06-06 02:40
The code for the popen* functions doesn't check the return
value of _fdopen, which will return null when the limit of
open stdio streams is reached.
A similar issue exists in the win32pipe module, which uses
almost exactly the same code.  See Pywin32 bug 1731778
( http://sourceforge.net/tracker/?func=detail&atid=551954&aid=1731778&group_id=78018 )
for more details.
msg59205 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-04 01:30
Modules/posixmodule.c:_PyPopen() needs to check for NULL after the
_fdopen() calls.
msg60178 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-19 14:48
An easy C task for the bug day.
msg114313 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-08-19 00:39
Can this be closed as popen2 is deprecated?
msg182508 - (view) Author: Ramchandra Apte (Ramchandra Apte) * Date: 2013-02-20 14:24
I think so.
msg185232 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2013-03-25 20:51
In particular, the popen2 entry says " "Deprecated since version 2.6: This module is obsolete.  Use the subprocess module.  Check especially the Replacing Older Functions with the subprocess Module section."

On core-mentorship list, Guido answered 'Sure'. [close it]

Let's hope that subprocess does better.
History
Date User Action Args
2013-03-25 20:51:34terry.reedysetstatus: open -> closed

nosy: + terry.reedy
messages: + msg185232

resolution: wont fix
stage: needs patch -> resolved
2013-02-20 14:24:30Ramchandra Aptesetnosy: + Ramchandra Apte
messages: + msg182508
2011-02-01 04:37:38belopolskysetnosy: frankty, rupole, christian.heimes, BreamoreBoy
stage: test needed -> needs patch
title: WinPython 2.3.3 crashes using popen2 to spawn lots of child -> popen2 on Windows does not check _fdopen return value
2010-08-19 00:39:11BreamoreBoysetnosy: + BreamoreBoy
messages: + msg114313
2010-05-11 20:46:04terry.reedysetversions: + Python 2.7
2009-02-14 11:31:39ajaksu2setstage: test needed
versions: - Python 2.5
2008-01-19 14:48:08christian.heimessetkeywords: + easy
type: crash
messages: + msg60178
components: + Interpreter Core
2008-01-04 01:30:45christian.heimessetpriority: normal -> high
nosy: + christian.heimes
messages: + msg59205
versions: + Python 2.6, Python 2.5, - Python 2.3
2004-01-28 20:28:33franktycreate