Author neologix
Recipients Aaron.Sherman, gregory.p.smith, neologix, pitrou, vstinner
Date 2011-02-25.10:48:37
SpamBayes Score 4.44644e-14
Marked as misclassified No
Message-id <1298630918.06.0.000134481049779.issue11314@psf.upfronthosting.co.za>
In-reply-to
Content
> If you stop to think about it, though, this is actually a shockingly huge percent increase. In any process creation scenario I'm familiar with, its overhead should be so small that you could bump it up several orders of magnitude and still not compete with executing a shell and asking it to do anything, even just exit.

os.popen just calls the popen(3) library call, which just performs a fork/execve and some dup/close in between.
subprocess.Popen is implemented in Python, so it doesn't come as a surprise that it's slower in your example.
But I think there's a point you're missing: you're not just creating a child process, but a shell.
subprocess is actually faster than popen to spawn executables (not scripts), because you don't have to spawn a shell.

For example:

$ cat /tmp/test.c 
int main(int argc, char *argv[])
{
        return 0;
}

$ cat /tmp/test_subprocess.py 
import subprocess

for i in range(10000):
    f = subprocess.Popen('/tmp/test')
    f.wait()

$ cat /tmp/test_popen.py 
import os

for i in range(10000):
    f = os.popen('/tmp/test')
    f.close()

$ time ./python /tmp/test_subprocess.py

real    0m13.933s
user    0m3.083s
sys     0m12.441s

$ time ./python /tmp/test_popen.py 

real    0m18.235s
user    0m4.293s
sys     0m15.176s

Given of important the subprocess overhead seems to you, I guess that the processes you're launching are not shell scripts, and thus you're probably better off using subprocess.
History
Date User Action Args
2011-02-25 10:48:38neologixsetrecipients: + neologix, gregory.p.smith, pitrou, vstinner, Aaron.Sherman
2011-02-25 10:48:38neologixsetmessageid: <1298630918.06.0.000134481049779.issue11314@psf.upfronthosting.co.za>
2011-02-25 10:48:37neologixlinkissue11314 messages
2011-02-25 10:48:37neologixcreate