Message205534
This is a spinoff of issue #19506: currently, subprocess.communicate() uses a 4K buffer when reading data from pipes.
This was probably optimal a couple years ago, but nowadays most operating systems have larger pipes (e.g. Linux has 64K), so we might be able to gain some performance by increasing this buffer size.
For example, here's a benchmark reading from a subprocess spawning "dd if=/dev/zero bs=1M count=100":
# before, 4K buffer
$ ./python ~/test_sub_read.py
2.72450800300021
# after, 64K buffer
$ ./python ~/test_sub_read.py
1.2509000449999803
The difference is impressive.
I'm attaching the benchmark script so that others can experiment a bit (on multi-core machines and also different OSes). |
|
Date |
User |
Action |
Args |
2013-12-08 09:38:10 | neologix | set | recipients:
+ neologix, gregory.p.smith, pitrou, vstinner, sbt, serhiy.storchaka |
2013-12-08 09:38:10 | neologix | set | messageid: <1386495490.95.0.0341161010002.issue19929@psf.upfronthosting.co.za> |
2013-12-08 09:38:10 | neologix | link | issue19929 messages |
2013-12-08 09:38:10 | neologix | create | |
|