I just discovered that the csv module's reader class in 3.x doesn't work
as expected when used as documented. The requirement has always been
that the CSV file is opened in binary mode so that embedded newlines in
fields are screwed up. Alas, in 3.x files opened in binary mode return
their contents as bytes, not unicode strings which are apparently not
allowed by the next() builtin:

% python3.1
Python 3.1a0 (py3k:70084M, Feb 28 2009, 20:46:48) 
[GCC 4.0.1 (Apple Inc. build 5490)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import csv    
>>> next(csv.reader(open("f.csv", "rb")))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
_csv.Error: iterator should return strings, not bytes (did you open the 
file in text mode?)
>>> next(csv.reader(open("f.csv", "r")))
['col1', 'col2', 'color']

At the very least the documentation for the csv.reader class is no
longer correct. However, I can't see how you can open a CSV file in
text mode and not screw up embedded newlines. I think binary mode
*has* to stay and some other way of dealing with bytes has to be found.

in _csv.c, the check is done here:

lineobj = PyIter_Next(self->input_iter);
   if (lineobj == NULL) {
      /* End of input OR exception */
      if (!PyErr_Occurred() && self->field_len != 0)
         PyErr_Format(error_obj,
            "newline inside string");
         return NULL;
      }
if (!PyUnicode_Check(lineobj)) {
   PyErr_Format(error_obj,
      "iterator should return strings, "
      "not %.200s "
      "(did you open the file in text mode?)",
      lineobj->ob_type->tp_name
   );
   Py_DECREF(lineobj);
   return NULL;
}

So the returned lineobj is a bytes type and then the PyUnicode_Check
throws the error.

Hi Skip,

Currently, once we are sure the lineobj is a unicode obj we then
get it's internal buffer using:

line = PyUnicode_AsUnicode(lineobj); 

for the purpose of iterating through the line.

is there an opportunity to use:

line = PyBytes_AsString(lineobj); 

(or similar approach if I have quoted an incorrect function) for the
case that we have a bytes object (not Unicode)?

This is in effect a duplicate of bpo-4847.

Summary:
The docs are CORRECT.
The 3.X implementation is WRONG.
The 2.X implementation is CORRECT.

See examples in my comment on bpo-4847.

Jervis> So the returned lineobj is a bytes type and then the
Jervis> PyUnicode_Check throws the error.

Right, but given that fact how do you get a Unicode string out of the bytes
without an encoding? You can't open a file in binary mode and give the
encoding arg.

John> The docs are CORRECT.
John> The 3.X implementation is WRONG.
John> The 2.X implementation is CORRECT.

I agree. I posted a note to python-dev referencing both tickets. Hopefully
one of the bytes/unicode experts there can shed some light on a possible
solution.

Skip

Setting bpo-4847 as superseder.