The following code from the documentation fails:
#################
import csv
reader = csv.reader(open("eggs.csv", "rb"))
for row in reader:
    print(row)
#####################
The output is:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 3, in <module>
_csv.Error: iterator should return strings, not bytes (did you open the
file in text mode?)


It works as expected in python 2.6

Do you expect to be able to read CSV as bytes or just to fix the 
documentation example?

In an unlikely scenario:
say one of the fields has an embedded \r. For instance "blahblah\r" is the value of the first column. Now open this file in text mode. What happens to this '\r' even before csv.reader sees it? If it remains intact, no problem. If it is converted to a newline in any platform, and that newline is not exactly \r, does this not introduce a problem?

Just curious...

Thanks.

----- Original Message ----
From: STINNER Victor <report@bugs.python.org>
To: jaywalkie@yahoo.com
Sent: Monday, January 5, 2009 12:05:59 PM
Subject: [issue4847] csv fails when file is opened in binary mode

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

Do you expect to be able to read CSV as bytes or just to fix the 
documentation example?

----------
nosy: +haypo

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4847>
_______________________________________

> say one of the fields has an embedded \r. For instance "blahblah\r" is the
> value of the first column. Now open this file in text mode. What happens to
> this '\r' even before csv.reader sees it?

I used rarely the CSV format, but it sounds strange to have a newline 
character in a column. Newlines characters (\r and \n) are reserved to mark 
the end of the line. Can you produce such file to test? :-)

I guess that the csv modules does something like readline().split(";").

make it '\r\n', if you want. Such files can be easily generated on windows text editors, or even linux ones nowadays. Upon reading, if the file is opened in text mode, this will probably be converted to \n even on linux by python (I may be wrong). Thus, \r gets lost.

Not a biggie, but you never know ;0)

----- Original Message ----
From: STINNER Victor <report@bugs.python.org>
To: jaywalkie@yahoo.com
Sent: Monday, January 5, 2009 12:23:45 PM
Subject: [issue4847] csv fails when file is opened in binary mode

STINNER Victor <victor.stinner@haypocalc.com> added the comment:

> say one of the fields has an embedded \r. For instance "blahblah\r" is the
> value of the first column. Now open this file in text mode. What happens to
> this '\r' even before csv.reader sees it?

I used rarely the CSV format, but it sounds strange to have a newline 
character in a column. Newlines characters (\r and \n) are reserved to mark 
the end of the line. Can you produce such file to test? :-)

I guess that the csv modules does something like readline().split(";").

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4847>
_______________________________________

You can avoid the newline translation problem by using the newline
parameter in open(). Set it to '' (the empty string) and any CR and LF
characters should remain intact.

As for the original problem, IMHO it is a documentation bug.

I think what you suggest makes most sense. 
Thanks.

----- Original Message ----
From: Antoine Pitrou <report@bugs.python.org>
To: jaywalkie@yahoo.com
Sent: Monday, January 5, 2009 12:47:21 PM
Subject: [issue4847] csv fails when file is opened in binary mode

Antoine Pitrou <pitrou@free.fr> added the comment:

You can avoid the newline translation problem by using the newline
parameter in open(). Set it to '' (the empty string) and any CR and LF
characters should remain intact.

As for the original problem, IMHO it is a documentation bug.

----------
assignee:  -> georg.brandl
components: +Documentation -Library (Lib)
nosy: +georg.brandl, pitrou
priority:  -> normal

_______________________________________
Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue4847>
_______________________________________

Short patch fixing the examples + the description of reader() and 
writer() (remove the "b" flag sentence: it's wrong, we need unicode!). 
I hope that I didn't break the documentation (i tried "make text" and 
i didn't get any warning).

Sorry, folks, we've got an understanding problem here. CSV files are
typically NOT created by text editors. They are created e.g. by "save as
csv" from a spreadsheet program, or as an output option by some database
query program. They can have just about any character in a field,
including \r and \n. Fields containing those characters should be quoted
(just like a comma) by the csv file producer. A csv reader should be
capable of reproducing the original field division. Here for example is
a dump of a little file I just created using Excel 2003:

C:\devel\csv>\python26\python -c "print repr(open('book1.csv','rb').read())"
'Field1,"Field 2 has a\nvery long\nheading",Field3\r\n1.11,2.22,3.33\r\n'

Inserting \n into a text field in Excel (using Alt-Enter) is a
well-known user trick.

Here's what we get from Python 2.6.1:
C:\devel\csv>\python26\python -c "import csv; print
repr(list(csv.reader(open('book1.csv','rb'))))"
[['Field1', 'Field 2 has a\nvery long\nheading', 'Field3'], ['1.11',
'2.22', '3.33']]
and the same by design all the way back to Python 2.3's csv module and
its ancestor, the ObjectCraft csv module.

However with Python 3.0.1 we get:
C:\devel\csv>\python30\python -c "import csv;
print(repr(list(csv.reader(open('book1.csv','rb')))))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
_csv.Error: iterator should return strings, not bytes (did you open the
file in text mode?)

This sentence in the documentation is NOT an error: """If csvfile is a
file object, it must be opened with the ‘b’ flag on platforms where that
makes a difference."""

The problem *IS* a "biggie".

This paragraph in the documentation (evidently introduced in 2.5) is
rather confusing:"""The parser is quite strict with respect to
multi-line quoted fields. Previously, if a line ended within a quoted
field without a terminating newline character, a newline would be
inserted into the returned field. This behavior caused problems when
reading files which contained carriage return characters within fields.
The behavior was changed to return the field without inserting newlines.
As a consequence, if newlines embedded within fields are important, the
input should be split into lines in a manner which preserves the newline
characters.""" Some examples of what it is talking about would be a very
good idea.

Hi all,

This patch takes the approach of assuming utf-8 format encoding
for files opened with 'rb' directive. 

That is:

1. Check if each line is Unicode Or Bytes Type.
2. If Bytes, get char array reference to internal buffer.
3. use PyUnicode_FromString to create a new unicode object from the
char* - This step assumes UTF-8.
4. get a Py_UNICODE reference to internal unicode object buffer and 
   continue as before.

Is this in the right direction at all?

Cheers,

Jervis

Before patching, could we discuss the requirements?

There are two different concepts:
(1) "text" file (assume that CR and/or LF are line terminators, and
provide methods for accessing a line at a time) versus "binary" file (no
such assumptions, no such access)
(2) reading the file as a raw undecoded "bytes" file or as a decoded
"str" file.

Options for 3.X:
(1) caller uses mode 'rb', is given bytes objects back.
(2) caller uses mode 'rt' and provides an encoding, is given str objects
back.
IMPORTANT: Option 2 must NOT not read the file as a collection of
"lines"; it must process it (conceptually at least) a character at a
time so that embedded CR and/or LF are not taken to be row terminators.

Following the line that 3.X line should do what's best, not what we used
to do, the implication is that we choose option 2.

... and it looks like Option 2 might already *almost* be in place.
Continuing with the previous example (book1.csv has embedded lone LFs):

C:\devel\csv>\python30\python -c "import csv;
print(repr(list(csv.reader(open('book1.csv','rt', encoding='ascii')))))"
[['Field1', 'Field 2 has a\nvery long\nheading', 'Field3'], ['1.11',
'2.22', '3.33']]

Looks good. However consider book2.csv which has embedded CRLFs:
C:\devel\csv>\python30\python -c "print(repr(open('book2.csv',
'rb').read()))"
b'Field1,"Field 2 has a\r\nvery
long\r\nheading",Field3\r\n1.11,2.22,3.33\r\n'

This gives:
C:\devel\csv>\python30\python -c "import csv;
print(repr(list(csv.reader(open('book2.csv','rt', encoding='ascii')))))"
[['Field1', 'Field 2 has a\nvery long\nheading', 'Field3'], ['1.11',
'2.22', '3.33']]

Not good. It should preserve ALL characters in the field.

> Not good. It should preserve ALL characters in the field.

Please look at the doc for open() and io.TextIOWrapper. The `newline`
parameter defaults to None, which means universal newlines with newline
translation. Setting to '' (yes, the empty string) enables universal
newlines but disables newline translation (that it, it will split lines
on all of ['\n', '\r', '\r\n'], but will leave these newlines intact
rather than convert them to '\n').

However, I think csv should accept files opened in binary mode and be
able to deal with line endings itself. How am I supposed to know the
encoding of a CSV file? Surely Excel uses a defined, default encoding
when exporting to CSV... that knowledge should be embedded in the csv
module.

pitrou> Please look at the doc for open() and io.TextIOWrapper. The
`newline` parameter defaults to None, which means universal newlines
with newline translation. Setting to '' (yes, the empty string) enables
universal newlines but disables newline translation ...

I had already read it. I gave it a prize for "least intuitive arg in the
language". So you plan to use that, reading "lines" instead of blocks?
You'll still have to examine which CRs and LFs are embedded and which
are line terminators. You might just as well use f.read(BLOCKSZ) and
avoid having to insist that the user explicitly write ", newline=''".

pitrou> However, I think csv should accept files opened in binary mode
and be able to deal with line endings itself. How am I supposed to know
the encoding of a CSV file? Surely Excel uses a defined, default
encoding when exporting to CSV... that knowledge should be embedded in
the csv module.

Excel has no default, because the user has no option -- the defined
encoding is "cp" + str(codepage_number_derived_from_locale), e.g.
"cp1252". Likewise other software writing delimited data to text files
will use (one of) the local legacy encoding(s).

So: (i) mode='rb' and no encoding => caller gets bytes back and needs to
do own decoding or (ii) mode='rb' and an encoding [which looks rather
daft and is currently not possible] and the the caller gets str objects.
Both of these are ugly -- hence my preference for the mode="rt" variety
of solution. Do we really want the double hassle of both a str csv
implementation and a bytes csv implementation?

> I had already read it. I gave it a prize for "least intuitive arg in the
> language".

Please open a bug, then :)

> So you plan to use that, reading "lines" instead of blocks?
> You'll still have to examine which CRs and LFs are embedded and which
> are line terminators. You might just as well use f.read(BLOCKSZ) and
> avoid having to insist that the user explicitly write ", newline=''".

Sorry, but who is "you" in that paragraph?
The csv module currently accepts any iterator yielding lines of text,
not only file objects. Changing this would be a major compatibility
break.

> Excel has no default, because the user has no option -- the defined
> encoding is "cp" + str(codepage_number_derived_from_locale), e.g.
> "cp1252".

Then Excel-generated CSV files all use different encodings? Gasp :-(

John> Options for 3.X:
    John> (1) caller uses mode 'rb', is given bytes objects back.
    John> (2) caller uses mode 'rt' and provides an encoding, is given str
    John>     objects back.

I believe #1 is the only correct option.  If you open a file in text mode
the low-level file reading code will unify all line endings to \n.  I don't
believe it's possible to override that behavior.

Skip

Antoine> However, I think csv should accept files opened in binary mode
    Antoine> and be able to deal with line endings itself. How am I supposed
    Antoine> to know the encoding of a CSV file? Surely Excel uses a
    Antoine> defined, default encoding when exporting to CSV... that
    Antoine> knowledge should be embedded in the csv module.

In fact, the csv module actually requires files to be opened in binary mode
but doesn't enforce that.  That it works in all but a few corner cases is
because so few CSV files contain fields containing embedded newlines.

Why doesn't open() allow you to specify an encoding when opening in binary
mode?  Even though it would be unused by the File object, it would serve as
an annotation for code using that open File object so they can do the
appropriate bytes-to-unicode conversion.

Skip

Antoine> .... The csv module currently accepts any iterator yielding
    Antoine> lines of text, not only file objects....

If the iterator yields bytes and had an encoding attribute that would serve
to allow the csv reader to perform the necessary conversion.  It would be
incumbent upon the user to specify the encoding.

Skip

[newlines translation]
> I don't
> believe it's possible to override that behavior.

It is, see my comments above.

> Why doesn't open() allow you to specify an encoding when opening in binary
> mode?

It would be more logical to pass the encoding argument to the csv reader
object instead.

Antoine> It would be more logical to pass the encoding argument to the
    Antoine> csv reader object instead.

What should be the default?

me> What should be the default?

Scratch that.  If the iterator passed to csv.reader is in a mode which will
cause it to emit bytes instead of unicode objects the caller must give an
encoding.  The csv.reader code will then perform the necessary
bytes-to-unicode conversion.  If bytes are returned by the iterator but no
encoding was given it should raise an exception (something standard?
something new?).

Skip

This issue seems to have simply been overlooked when 3.0 was released.
It should be fixed in the next round of 3.0 and 3.1 updates.  Any
feeback on the idea that the csv.reader constructor (and probably the
DictReader and proposed NamedTupleReader constructors) should take an
optional encoding argument?  In fact, the writers should as well
which would inform the writer how to encode the output when writing.

I think it's good if it allowed passing in a binary file and an
encoding, but I think it would be crazy if it wouldn't also take a text
file.  After all the primary purpose of a CSV file, edge cases
notwithstanding, is to look like text to the end user in a text editor
(as opposed to Excel spreadsheets, which look like binary gobbledygook
unless opened with Excel).

Are there any use cases for returning bytes strings instead of text
strings?  If so that should probably be another flag.

I've added some unit tests for embedded newlines, and py3k csv passes
(on linux at least) when newline='' is used.  Unless someone can provide
a test case that fails when newline='' is used, I propose we fix the
documentation and leave the code alone.

I'm attaching a proposed doc patch for comment.  I replace mentions of
'rb' with "newline=''", including in the examples.  I also deleted the
unicode discussion (since CSV obviously handles unicode now) as well as
the extensive unicode examples that are no longer relevant.  (There are
a couple other tweaks I made as I went along, but nothing substantial.)

David> I've added some unit tests for embedded newlines, and py3k csv
    David> passes (on linux at least) when newline='' is used.  Unless
    David> someone can provide a test case that fails when newline='' is
    David> used, I propose we fix the documentation and leave the code
    David> alone.

This thread is getting a bit long.  Can someone summarize how the expected
usage of the csv module is supposed to change?  If I read things correctly,
instead of requiring (in the general case) that csv files be opened in
binary mode, the requirement will be that they be opened with newline=''.
This will thwart any attempts by the io module at newline translation, but
since the file is still opened in text mode its contents will implicitly be
Unicode (or Unicode translated to bytes with a specific encoding).  That
encoding will also be specified in the call to open().

Is this about correct?  Do any test cases need to be updated or added?  I
notice that something called BytesIO is imported from io but not used.  Were
some test cases removed which used to involve that class or is that a 2to3
artifact?

Skip

David> I also deleted the unicode discussion (since CSV obviously
    David> handles unicode now) ...

Maybe there should be a simple example showing use of the encoding parameter
to open() to encode Unicode on write and decode to Unicode on read?

Skip

On Thu, 2 Apr 2009 at 17:55, Skip Montanaro wrote:
> This thread is getting a bit long.  Can someone summarize how the expected
> usage of the csv module is supposed to change?  If I read things correctly,
> instead of requiring (in the general case) that csv files be opened in
> binary mode, the requirement will be that they be opened with newline=''.
> This will thwart any attempts by the io module at newline translation, but
> since the file is still opened in text mode its contents will implicitly be
> Unicode (or Unicode translated to bytes with a specific encoding).  That
> encoding will also be specified in the call to open().

I believe that is an accurate summary.

> Is this about correct?  Do any test cases need to be updated or added?  I
> notice that something called BytesIO is imported from io but not used.  Were
> some test cases removed which used to involve that class or is that a 2to3
> artifact?

I will look in to this, and add an encoding example to the docs as you
suggest in another email.

--David

So this is a doc bug? If so, need it still block tomorrow's release?

Nope.  Sorry I forgot to change the priority.

> Is this about correct?  Do any test cases need to be updated or added?  I
> notice that something called BytesIO is imported from io but not used.
 Were
> some test cases removed which used to involve that class or is that a 2to3
> artifact?

The set of test cases is almost exactly parallel between 2.7 and 3.1. 
3.1 adds an additional DictReader test, and refactors one set of tests
to make the code simpler.  Other than that, it is just a matter of
replacing the file opening and closing code that uses binary mode with
'with' statements where the open uses newline=''.  BytesIO is not used
so would appear to be a conversion artifact.

Oh, yeah, other 3.1 differences are that the unicode test is uncommented
and updated, and a test is added to make sure nulls are handled correctly.

Doc patch applied in r71116.