Author gregory.p.smith
Recipients Yonatan Goldschmidt, gregory.p.smith, izbyshev, koobs, pablogsal, ronaldoussoren
Date 2020-10-25.06:38:16
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1603607896.72.0.37134453627.issue35823@roundup.psfhosted.org>
In-reply-to
Content
Performance improvement measured on a 1.4Ghz quad aarch64 A57 (nvidia jetson nano):

#define VFORK_USABLE 1
 test_subprocess: 36.5 seconds

#undef VFORK_USABLE
 test_subprocess: 45 seconds

Nice.  I really didn't expect a 20% speedup on its testsuite alone!

Lets dive into that with a microbenchmark:

$ ./build-novfork/python ten_seconds_of_truth.py
Launching /bin/true for 10 seconds in a loop.
Launched 2713 subprocesses in 10.00194378796732 seconds.
271.247275281014 subprocesses/second.
Increased our mapped pages by 778240KiB.
Launching /bin/true for 10 seconds in a loop.
Launched 212 subprocesses in 10.006392606999725 seconds.
21.186456331095847 subprocesses/second.

$ ./build/python ten_seconds_of_truth.py
Launching /bin/true for 10 seconds in a loop.
Launched 3310 subprocesses in 10.001623224001378 seconds.
330.94628000551285 subprocesses/second.
Increased our mapped pages by 778240KiB.
Launching /bin/true for 10 seconds in a loop.
Launched 3312 subprocesses in 10.001519071985967 seconds.
331.1496959773679 subprocesses/second.

Demonstrating perfectly the benefit of vfork().  The more mapped pages, the slower regular fork() becomes.  With vfork() the size of the process's memory map does not matter.

ten_seconds_of_truth.py:

```python
from time import monotonic as now
import subprocess

def benchmark_for_ten():
    print('Launching /bin/true for 10 seconds in a loop.')

    count = 0
    start = now()
    while now() - start < 10.:
        subprocess.run(['/bin/true'])
        count += 1
    end = now()
    duration = end-start

    print(f'Launched {count} subprocesses in {duration} seconds.')
    print(f'{count/duration} subprocesses/second.')

benchmark_for_ten()

map_a_bunch_of_pages = '4agkahglahaa^#\0ag3\3'*1024*1024*40

print(f'Increased our mapped pages by {len(map_a_bunch_of_pages)//1024}KiB.')

benchmark_for_ten()
```
History
Date User Action Args
2020-10-25 06:38:16gregory.p.smithsetrecipients: + gregory.p.smith, ronaldoussoren, koobs, izbyshev, pablogsal, Yonatan Goldschmidt
2020-10-25 06:38:16gregory.p.smithsetmessageid: <1603607896.72.0.37134453627.issue35823@roundup.psfhosted.org>
2020-10-25 06:38:16gregory.p.smithlinkissue35823 messages
2020-10-25 06:38:16gregory.p.smithcreate