Author reacocard
Recipients reacocard
Date 2008-04-07.20:55:09
SpamBayes Score 0.159611
Marked as misclassified No
Message-id <1207601711.27.0.24453345668.issue2576@psf.upfronthosting.co.za>
In-reply-to
Content
This is a reposting of issue 508157, as requested by gvanrossum.

The socket file object in httplib is opened without any buffering
resulting in very slow performance of read().  The specific problem is
in the httplib.HTTPResponse constructor.  It calls sock.makefile() with
a 0 for the buffer size.  changing the buffer size to 4096 improved the
time needed to download 10MB from 15.5s to 1.78s, almost 9x faster.
Repeat downloads of the same file (meaning the server now has the file
cached in memory), yield times of 15.5s and 0.03s, a 500x improvement.
When fetching from a server on the local network, rather than from
localhost, these times become 15.5s and 0.9s in both cases, a 17x
speedup. Real-world situations will likely be a mix of these, however it
is safe to say the speed improvement will be substantial. Adding an
option to adjust the buffer size would be very welcome, though the
default value should still be zero, to avoid the issues already
mentioned in issue 508157.

These speed results were obtained with python2.5 and apache2 under
Ubuntu linux, using the code found here: http://pastebin.ca/973578
History
Date User Action Args
2008-04-07 20:55:11reacocardsetspambayes_score: 0.159611 -> 0.159611
recipients: + reacocard
2008-04-07 20:55:11reacocardsetspambayes_score: 0.159611 -> 0.159611
messageid: <1207601711.27.0.24453345668.issue2576@psf.upfronthosting.co.za>
2008-04-07 20:55:10reacocardlinkissue2576 messages
2008-04-07 20:55:10reacocardcreate