This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author jason.w.kim
Recipients jason.w.kim
Date 2007-10-05.20:09:28
SpamBayes Score 0.0017693822
Marked as misclassified No
Message-id <1191614970.62.0.0159920077283.issue1241@psf.upfronthosting.co.za>
In-reply-to
Content
Hi. 

I am currently using subprocess.py (2.4.4 and above) to try to have a
portable way of running a sub task in linux and windows.

I ran into a strange problem - a program runs and is "timed out"
but the the subprocess's stdout and stderr are not fully "grabbed"
So I've been trying various ways to force "flush" the stdout and stderr
of the child process so that at least partial results can be saved.

The "problem" app being spawned off is very simple:
--------------------------------------
#include <stdio.h>

int main() {
  int i = 0;
  for (i = 0; i < 1000; ++i) {
    printf("STDOUT boo %d\n",i);
    fprintf(stdout,"STDOUT sleeping %d\n",i);
    fprintf(stderr,"STDERR sleeping %d\n",i);
    //fflush(stdout);
    //fflush(stderr);
    sleep(1);
  } 
}

-----------------------------------------------

i.e. it just dumps its output to both stdout and stderr. The issue that
I am seeing is that no matter what options I tried to place for
subprocess(), the ONLY output I see from the executed process are

"STDERR sleeping " lines, UNLESS I uncomment the fflush(stdout) line in
the application.

Executing the script with python -u doesn't seem to help either.
Now, if the task completes normally, then I am able to grab the entire
stdout and stderr produced by the subprocess. The issue is that I can't
seem to grab the partial output for stdout, and there does not seem to
be a way to make the file descriptors returned by pipe() to be unbuffered.

So the question is: what is the preferred method of forcing the pipe()
file descriptors created by subprocess.__init__() to be fully unbuffered?

Second, is there a better way of doing this?
i.e. a portable way to spawn off a task, with an optional timeout, grab
any partial results from the task's stdout and stderr, and grab the
return code from the child task?

Any hints and advice will be greatly appreciated.

Thank you.

The relevant snippet of python code is:

import threading
from signal import *
from subprocess import *
import time
import string
import copy
import re
import sys
import os
from glob import glob
from os import path
import thread

class task_wrapper():
   def run(s):
      if s.timeout > 0:
        #print "starting timer for ",s.timeout
        s.task_timer = threading.Timer(s.timeout, task_wrapper.cleanup, [s])
        s.task_timer.start()
      s.task_start_time = time.time()
      s.task_end_time = s.task_start_time
      s.subtask=Popen(s.cmd, bufsize=0, env=s.env, stdout=PIPE, stderr=PIPE)
      s.task_out, s.task_err = s.subtask.communicate()


  def kill(s, subtask):
    """ attempts a portable way to kill things
    First, flush the buffer
    """
    print "killing", subtask.pid
    sys.stdout.flush()
    #s.subtask.stdin.flush()
    print "s.subtask.stdout.fileno()=",s.subtask.stdout.fileno()
    print "s.subtask.stderr.fileno()=",s.subtask.stderr.fileno()
    #os.fsync(s.subtask.stderr.fileno())
    #os.fsync(s.subtask.stdout.fileno())    
    s.subtask.stdout.flush()
    s.subtask.stderr.flush()
    
    if os.name == "posix":
      os.kill(subtask.pid, SIGKILL)
    elif os.name == "nt":
      import win32api
      win32api.TerminateProcess(subtask._handle ,9)

  def cleanup(s, mode="TIMEOUT"):
    s.timer_lock.acquire()
    if s.task_result == None:
    if mode == "TIMEOUT":
       s.msg( """ Uhoh, subtask took too long""")
       s.kill(s.subtask) 
       ....
History
Date User Action Args
2007-10-05 20:09:30jason.w.kimsetspambayes_score: 0.00176938 -> 0.0017693822
recipients: + jason.w.kim
2007-10-05 20:09:30jason.w.kimsetspambayes_score: 0.00176938 -> 0.00176938
messageid: <1191614970.62.0.0159920077283.issue1241@psf.upfronthosting.co.za>
2007-10-05 20:09:30jason.w.kimlinkissue1241 messages
2007-10-05 20:09:28jason.w.kimcreate