hi,  we use python for the Dark Energy Survey and use the subprocess module to run a variety of programs, which we do not control.  One concern we have is to monitor system information such as whether the code causes swapping.  It's be really useful if

-- The Pipe object (and similar objectsI in the subprocess module could ....

- have waits in unix implemented by  os.wait4() for unix, instead of os.waitpid()
- have an new data member, rusage, make the resource usage available to the caller.

An additional comment is that all of the wait() family of system calls are implemented in terms of wait4, and there is an argument that hiding usage has no effective cost.  

There is a wait(self,...)  method in pipe. it seems a simple change to subprocess.Pipe to call os.wait4 in the unix variant and stash usage in a data member.

I guess that you mean "Popen" when you write "Pipe"?

Even if Popen has a high-level wait() method implemented with os.waitpid(), you are free to use a low-level function using the pid attribute.

proc = subprocess.Popen(...)
os.wait4(proc.pid, ...)

> have an new data member, rusage, make the resource usage available to the caller.

I don't know how to get the rusage of a specific child process, but you can use resource.getrusage(resource.RUSAGE_CHILDREN) which gives the usage of *all* child processes. Again, you have the pid, and so you are free to use any function to retrieve the resource usage of the child process. See also this project which can help you:
https://pypi.python.org/pypi/psutil

I don't think that the subprocess should be modified to your use case, it's already possible to implement you use cases without modify it. Python is a (very) portable language, and it's very hard to provide the same API for such low-level metrics (rusage). I don't think that Windows provides exactly the same data for example.

You can build your own module on top of subprocess and other modules like psutil.

Victor, thanks so much for reading my ticket and contacting me.

I prototyped what makes sense to me by subclassing subprocess.Popen and
Over-ridden the wait method to do what I suggested. Proof of concept
attached.  \
It¹s  a bit of a hack since I cribbed code from Popen.wait().

Now that you¹ve made me think harder,  The use case that stood out was the
case 
where I called the communicate() method which both handles IO and the
termination 
of the process AFAICT.

I _do_ understand the portability argument you are making (and am clueless
about
what¹s to be done with windows).

 ‹ Thanks
‹  Don

On 5/14/14, 5:28 PM, "STINNER Victor" <report@bugs.python.org> wrote:

>
>STINNER Victor added the comment:
>
>I guess that you mean "Popen" when you write "Pipe"?
>
>Even if Popen has a high-level wait() method implemented with
>os.waitpid(), you are free to use a low-level function using the pid
>attribute.
>
>proc = subprocess.Popen(...)
>os.wait4(proc.pid, ...)
>
>> have an new data member, rusage, make the resource usage available to
>>the caller.
>
>I don't know how to get the rusage of a specific child process, but you
>can use resource.getrusage(resource.RUSAGE_CHILDREN) which gives the
>usage of *all* child processes. Again, you have the pid, and so you are
>free to use any function to retrieve the resource usage of the child
>process. See also this project which can help you:
>https://pypi.python.org/pypi/psutil
>
>I don't think that the subprocess should be modified to your use case,
>it's already possible to implement you use cases without modify it.
>Python is a (very) portable language, and it's very hard to provide the
>same API for such low-level metrics (rusage). I don't think that Windows
>provides exactly the same data for example.
>
>You can build your own module on top of subprocess and other modules like
>psutil.
>
>----------
>nosy: +haypo
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue21504>
>_______________________________________
>

I think the answer is that if you want that level of control you can't use communicate, you have to implement what you specifically need.

I'm going to close this as rejected.  It could be reopened if someone can find a way to propose something that makes sense cross-platform.

Ok.  I have a subclass with a copied out - then-modified wait method.
 ¹twill do for now.
I¹m clueless as to what the windows analog of wait4() would be.

On 5/30/14, 3:10 PM, "R. David Murray" <report@bugs.python.org> wrote:

>
>R. David Murray added the comment:
>
>I think the answer is that if you want that level of control you can't
>use communicate, you have to implement what you specifically need.
>
>I'm going to close this as rejected.  It could be reopened if someone can
>find a way to propose something that makes sense cross-platform.
>
>----------
>nosy: +r.david.murray
>resolution:  -> rejected
>stage:  -> resolved
>status: open -> closed
>
>_______________________________________
>Python tracker <report@bugs.python.org>
><http://bugs.python.org/issue21504>
>_______________________________________
>