Author vstinner
Recipients gregory.p.smith, izbyshev, nanjekyejoannah, vstinner
Date 2018-12-20.09:35:23
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1545298524.44.0.788709270274.issue35537@psf.upfronthosting.co.za>
In-reply-to
Content
"posix_spawn() is faster"

This assumption needs a benchmark!

I ran a benchmark on Linux (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28) using attached subprocess_bench.py.

I rebased the PR 11242 on master and run the benchmark in virtual environment:

git co pr/11242
# current branch: PR 11242
make distclean
./configure
make
./python -m venv env
env/bin/python -m pip install perf
env/bin/python subprocess_bench.py -v -o posix_spawn.json
git co master
env/bin/python subprocess_bench.py -v -o fork_exec.json
env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 

The result is quite explicit: the PR makes subprocess.Popen 61x faster!!! 

$ env/bin/python -m perf compare_to fork_exec.json posix_spawn.json 
Mean +- std dev: [fork_exec] 27.1 ms +- 0.4 ms -> [posix_spawn] 447 us +- 163 us: 60.55x faster (-98%)

That's the best case:

* The parent process (Python) allocated 2 GiB of memory: that's not uncommon for large application. On OpenStack for example, it's common that a process takes more than 1 GiB.
* The child process has a short execution time and allocates few memory.


On my config (Fedora 29, Linux kernel 4.19.9-300.fc29.x86_64, glibc 2.28), os.posix_spawn() uses vfork():

$ strace -o trace ./python -c 'import subprocess; subprocess.run(["/bin/true"], close_fds=False, restore_signals=False)'
$ grep clone trace
clone(child_stack=0x7fab28ac0ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = 23073

I guess that the 61x speedup mostly comes from vfork().

See also bpo-34663 for previous discussion about vfork() and os.posix_spawn(). In short, the glibc is smart and detects when vfork() can be used or not.
History
Date User Action Args
2018-12-20 09:35:24vstinnersetrecipients: + vstinner, gregory.p.smith, izbyshev, nanjekyejoannah
2018-12-20 09:35:24vstinnersetmessageid: <1545298524.44.0.788709270274.issue35537@psf.upfronthosting.co.za>
2018-12-20 09:35:24vstinnerlinkissue35537 messages
2018-12-20 09:35:23vstinnercreate