classification
Title: subprocess Popen wait() function hangs when stdout is > 20480
Type: behavior Stage:
Components: None Versions: Python 2.4
process
Status: closed Resolution: wont fix
Dependencies: Superseder: Doc: subprocess wait() may lead to dead lock
View: 1606
Assigned To: Nosy List: Vano, christian.heimes, draghuram, superwesman
Priority: normal Keywords:

Created on 2007-10-10 17:41 by superwesman, last changed 2008-01-06 14:12 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
brent.py superwesman, 2007-10-10 17:41
Messages (4)
msg56317 - (view) Author: superwesman (superwesman) Date: 2007-10-10 17:41
hi - I noticed some odd behaviour with subprocess when the output
(stdout) is large - I did some sleuthing and found a reproduceable case
- I tested this on solaris using python 2.4.2 (sparc) and using 2.5
(opteron) - I've attached a python script that details the problem - I
created 2 files ("small_file" and "large_file") that are 20480 and 20481
bytes, respectively ....

> ls -lrt ./tmp/*_file
-rw-r--r--   1 wtorres  staff      20481 Oct 10 12:32 ./tmp/large_file
-rw-r--r--   1 wtorres  staff      20480 Oct 10 12:32 ./tmp/small_file

the script cats each file, thus generating an appropriately sized stdout
- the workaround we came up with was to use our own file handle and use
the parameter stdout=my_file_handle.fileno() instead of stdout=PIPE -
you can see this in the attached example as well

> python ./tmp/brent.py 
testing with a small file - these should both work
using my own file handle .... ok
using PIPE ... ok
testing with a large file - the second one here should fail
using my own file handle .... ok
using PIPE ...
Traceback (most recent call last):
  File "./tmp/brent.py", line 43, in ?
    test_popen_bug( large_file )
  File "./tmp/brent.py", line 31, in test_popen_bug
    rc = my_popen.wait()
  File "/tools/python/2.4.2/solaris/lib/python2.4/subprocess.py", line
1007, in wait
    pid, sts = os.waitpid(self.pid, 0)
KeyboardInterrupt

based on all this, it seems that the PIPE file handle has some buffer or
something that is filling up and not allowing us to continue... or
something.  Thoughts?
-w
msg56682 - (view) Author: Raghuram Devarakonda (draghuram) (Python triager) Date: 2007-10-23 16:58
PIPE on POSIX are implemented using pipe system call so it will have all
the limitations of the system buffering that is used for implementing
pipes. The buffer size is system dependent. For example, on my Linux, I
hit the hang condition at 64K and not 20480 bytes (as is your case).
What basically happens is that the child fills up the pipe buffer and is
blocked until space is available. Since the parent in your test script
is not reading the bytes, space will never be freed up. 

I suggest that the bug is closed as invalid.
msg57794 - (view) Author: Ivan Pozdeev (Vano) Date: 2007-11-23 18:13
I suggest adding this as a note to the subprocess module documentation instead. The "17.1.2 Popen Objects" section seems to be the right place.
msg59371 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-01-06 14:12
The docs should be updated, see #1606
History
Date User Action Args
2008-01-06 14:12:46christian.heimessetstatus: open -> closed
resolution: wont fix
superseder: Doc: subprocess wait() may lead to dead lock
messages: + msg59371
nosy: + christian.heimes
2007-11-23 18:13:26Vanosetnosy: + Vano
messages: + msg57794
2007-10-23 16:58:45draghuramsetnosy: + draghuram
messages: + msg56682
2007-10-10 17:41:49superwesmancreate