Problem in the newline handling in io.py, class
IncrementalNewlineDecoder, method decode. It reads text files in 128-
byte chunks. Converting CR LF to \n requires special case handling
when '\r' is detected at the end of the decoded chunk in case
there's an LF at the start of the next chunk. It prepends b'\r' (only 1
byte) to the next chunk's raw bytes and decodes that. But \r in UTF-16
takes 2 bytes; we are now 1 byte out of kilter and various failures are
possible (including silently producing garbage output from a truncated
file with an odd number of bytes).

The attached script illustrates the problems.

The bug is in IncrementalNewlineDecoder, not in the codec nor 
TextIOWrapper.

Smaller example to demonstrate the problem.

Here is a patch for test_io.py: check the problem by adding new 
encodings to TextIOWrapperTest.testNewlines().

Ugly patch to fix this issue:
 - add more regression tests for charsets UTF-16*, UTF-32*
 - add mandatory argument "encoding" to io.IncrementalNewlineDecoder 
constructor => BREAK THE API
 - use the encoding the encode "\r"
 - most ulgy hack: strip the BOM for codecs "UTF-16" and "UTF-32" (when 
encoding "\r" to bytes) => I don't know how to encode "\r" without the 
BOM

A couple of suggestions:

- if IncrementalNewlineDecoder gets an encoding argument, it can also
instantiate the decoder itself; that way the API is a bit simpler

- to encode '\r' without the BOM, you can e.g. use an incremental
encoder and encode it twice:
>>> enc = codecs.getincrementalencoder('utf16')('strict')
>>> enc.encode('\r')
b'\xff\xfe\r\x00'
>>> enc.encode('\r')
b'\r\x00'


I think breaking the API can be ok since the original API is broken
(witness this bug). There can even be a compatibility mode where we
check whether `encoding` has an encode() method, and if it has, take it
as the decoder object rather than the encoding name.

Here is a simpler patch with a different approach and a lot of tests.
The advantage is that it doesn't break the API.

This new variant also removes the dangerous hack in getstate / setstate.

utf16_newlines2.patch looks good to me.

This is a data corruption issue.  If it is deferred for 3.0.1 it must be 
fixed in 3.0.2.

+1 on putting this in 3.0.1.

Committed to py3k and release30-maint in r67760 and r67759. Needs
backporting to 2.x.

Backported to trunk and 2.6.2 in r67762 and r67764.