This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author kbdavidson
Date 2002-01-24.21:48:44
SpamBayes Score
Marked as misclassified
The socket file object underlying the return from 
urllib.urlopen() is opened without any buffering 
resulting in very slow performance of results.readline
().  The specific problem is in the 
httplib.HTTPResponse constructor.  It calls 
sock.makefile() with a 0 for the buffer size.  Forcing 
the buffer size to 4096 results in the time for 
calling readline() on a 60K character line to go from 
16 seconds to .27 seconds (there is other processing 
going on here but the magnitude of the difference is 

I am using Python 2.0 so I can not submit a patch 
easily but the problem appears to still be present in 
the 2.2 source.  The specific change is to change the 
0 in sock.makefile() to 4096 or some other reasonable 
buffer size:

class HTTPResponse:
    def __init__(self, sock, debuglevel=0):
        self.fp = sock.makefile('rb', 0)    <= change 
to 4096
        self.debuglevel = debuglevel

Date User Action Args
2007-08-23 13:58:52adminlinkissue508157 messages
2007-08-23 13:58:52admincreate