The Buffered I/O interface in the io module has the user specify buffer 
limits such as size and max_buffer_size.   The first limit (size) is 
easy to understand as a buffering threshold at which writes will occur.  
However, no apparent attempt is made to strictly limit the internal 
buffer size to max_buffer_size.
   
In BuffererWriter.write(), one of the first operations is 

     self._write_buf.extend(b)

which simply extends the buffer by the full data being written.  If b 
happens to be a large string (e.g., megabytes or even the entire 
contents of a big file), then the internal I/O buffer makes a complete 
copy of the data, effectively doubling the memory requirements for 
carrying out the write operation.

I suppose most programmers might not notice given that everyone has 
gigabytes of RAM these days, but you certainly don't see this kind of 
buffering behavior in the operating system kernel or in the C library.

Some patch suggestions (details left to the maintainers of this module):

1. Don't extend self._write_buf by more than the max_buffer_size.

     fragment = b[:self.max_buffer_size - len(self._write_buf)]
     self._write_buf.extend(fragment)

2. For large blocking writes, simply carry out the remaining I/O 
   operations in the write() method instead of in the _flush_locked()
   method.   Try to use the original input data b as the data
   source instead of making copies of it.  And if you have to copy
   the data, don't do it all at once.

I've attached my first attempt at fixing this as io-bufwrite-gps01.

Unfortunately it causes the Lib/test/test_io.py testThreads to fail:

======================================================================
FAIL: testThreads (__main__.BufferedReaderTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "Lib/test/test_io.py", line 456, in testThreads
    self.assertEqual(s.count(c), N)
AssertionError: 30 != 1000

I haven't read the test yet, but my guess is that the test is broken and
is blindly calling write() without checking the return value.

Yep, the test was ignoring the return value from BufferedWriter.write. 
Fixed in the attached io-bufwrite-gps02.

This can wait for 3.0.1 and 2.6.1/2 (depending on the 2.6.x release
schedule).

I do not agree. User code should not have to verify the return value of write().

When a big amount of data is passed to BufferedWriter.write,
- normal blocking streams should block until all data has been written.
(the implementation may call self.raw.write several times)
- non-blocking streams should raise BlockingIOError indicating the number of 
characters actually written.

This applies only to buffered files, of course.

I agree with Amuary, write() traditionally never writes less data unless
the underlying IO is in nonblocking mode.

I'm working up a new patch to write to self.raw in max_buffer_size
chunks with as few data copies as possible, raising an appropriate
BlockingIOError (as the surrounding code does) when the IO on the
underlying self.raw.write raises one through self._flush_unlocked().

Okay, here's a new patch that obeys the blocking vs nonblocking
semantics properly.  It still needs explicit unit tests for proper behavior.

According to msg76581, the patch fails review (lacking test cases), so
I'm removing the "needs review" keyword.

I agree with previous comments that write() should definitely write all 
data when in blocking mode.

Updated patch with unit tests to verify the write behavior attached in
-gps04.  Up for code review at http://codereview.appspot.com/12470/show

Hi!

that sounds
> like a behavior change.  I'd be fine with removing it for the 3.1/2.7
version of
> this code (though I hope people will be using the C implementation
instead).

Well, either it's supported and it will have to go through a deprecation
phase, or it's unsupported and it can be ripped out right now...

I don't think it should be supported at all, given that the semantics of
writing an iterable of ints are totally non-obvious. Reading both the
PEP and the docstrings in io.py, I only see mentions of "bytes" and
"buffer", not of an iterable of ints. Perhaps Guido should pronounce.

(do you know of any code relying on this behaviour? the C version
obviously does not support it and all regression tests pass fine, except
for an SSL bug I filed)

http://codereview.appspot.com/12470

I know of no code relying on this behavior, I merely duplicated the 
behavior that existed while writing the patch.  I'm happy to simplify.  
Adding Guido to the CC list for a pronouncement.

Guido is it okay to change io.BufferedWriter.write() in release30-maint 
to no longer accept iterables that do not support len() and things that 
do not support the buffer API?

Antoine already notes in the codereview that other existing io classes 
such as FileIO currently do not support non bytes/buffer or buffer-api 
supporting inputs:  http://codereview.appspot.com/12470/patch/1/3

@Gregory, that sounds like an odd enough use case to skip.  However you
might want to look for __length_hint__ before giving up?

OTOH unless the use case is real, why not support it but making it slow?

I've uploaded a new patch set with more extensive unit tests.  It also
handles the case of writing array.array objects (or anything with a
memoryview itemsize > 1).  The previous code would buffer by item rather
than by byte.  it has been updated in codereview as well.

fwiw, I decided Guido and Antoine were right and took out the support
for input that did not support len() to keep things a bit simpler.

max_buffer_size is no longer used, so this issue is obsolete ;)