This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author beazley
Recipients beazley
Date 2008-11-25.12:34:57
SpamBayes Score 1.6652429e-09
Marked as misclassified No
Message-id <1227616500.52.0.902847325707.issue4428@psf.upfronthosting.co.za>
In-reply-to
Content
The Buffered I/O interface in the io module has the user specify buffer 
limits such as size and max_buffer_size.   The first limit (size) is 
easy to understand as a buffering threshold at which writes will occur.  
However, no apparent attempt is made to strictly limit the internal 
buffer size to max_buffer_size.
   
In BuffererWriter.write(), one of the first operations is 

     self._write_buf.extend(b)

which simply extends the buffer by the full data being written.  If b 
happens to be a large string (e.g., megabytes or even the entire 
contents of a big file), then the internal I/O buffer makes a complete 
copy of the data, effectively doubling the memory requirements for 
carrying out the write operation.

I suppose most programmers might not notice given that everyone has 
gigabytes of RAM these days, but you certainly don't see this kind of 
buffering behavior in the operating system kernel or in the C library.

Some patch suggestions (details left to the maintainers of this module):

1. Don't extend self._write_buf by more than the max_buffer_size.

     fragment = b[:self.max_buffer_size - len(self._write_buf)]
     self._write_buf.extend(fragment)

2. For large blocking writes, simply carry out the remaining I/O 
   operations in the write() method instead of in the _flush_locked()
   method.   Try to use the original input data b as the data
   source instead of making copies of it.  And if you have to copy
   the data, don't do it all at once.
History
Date User Action Args
2008-11-25 12:35:00beazleysetrecipients: + beazley
2008-11-25 12:35:00beazleysetmessageid: <1227616500.52.0.902847325707.issue4428@psf.upfronthosting.co.za>
2008-11-25 12:34:58beazleylinkissue4428 messages
2008-11-25 12:34:57beazleycreate