Author neologix
Recipients gregory.p.smith, neologix, pitrou, sbt, serhiy.storchaka, vstinner
Date 2013-12-08.09:38:10
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1386495490.95.0.0341161010002.issue19929@psf.upfronthosting.co.za>
In-reply-to
Content
This is a spinoff of issue #19506: currently, subprocess.communicate() uses a 4K buffer when reading data from pipes.
This was probably optimal a couple years ago, but nowadays most operating systems have larger pipes (e.g. Linux has 64K), so we might be able to gain some performance by increasing this buffer size.

For example, here's a benchmark reading from a subprocess spawning "dd if=/dev/zero bs=1M count=100":

# before, 4K buffer
$ ./python ~/test_sub_read.py 
2.72450800300021

# after, 64K buffer
$ ./python ~/test_sub_read.py 
1.2509000449999803

The difference is impressive.

I'm attaching the benchmark script so that others can experiment a bit (on multi-core machines and also different OSes).
History
Date User Action Args
2013-12-08 09:38:10neologixsetrecipients: + neologix, gregory.p.smith, pitrou, vstinner, sbt, serhiy.storchaka
2013-12-08 09:38:10neologixsetmessageid: <1386495490.95.0.0341161010002.issue19929@psf.upfronthosting.co.za>
2013-12-08 09:38:10neologixlinkissue19929 messages
2013-12-08 09:38:10neologixcreate