classification
Title: csv.DictWriter can't handle extra non-string fields
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.4, Python 3.3, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: python-dev, r.david.murray, tomas_grahn, vajrasky
Priority: normal Keywords: patch

Created on 2013-10-30 10:59 by tomas_grahn, last changed 2013-11-19 18:28 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
csv-patch.diff tomas_grahn, 2013-10-30 10:59
fix_error_message_write_fields_not_in_fieldnames.patch vajrasky, 2013-11-12 08:59 review
Messages (8)
msg201727 - (view) Author: Tomas Grahn (tomas_grahn) Date: 2013-10-30 10:59
When csv.DictWriter.writerow is fed a dict with extra fieldnames (and extrasaction='raise') and any of those extra fieldnames aren't strings, a TypeError-exception is thrown. To fix the issue; in csv.py, edit the line:
raise ValueError("dict contains fields not in fieldnames: " + ", ".join(wrong_fields))

to:
raise ValueError("dict contains fields not in fieldnames: " + ", ".join(repr(wrong_field) for wrong_field in wrong_fields))

Attached is a patch that fixes the problem (works in both 2.6 and 2.7, I haven't tried anything else).

Here is a simple test to demonstrate the problem:

import cStringIO, csv

sio=cStringIO.StringIO()
writer=csv.DictWriter(sio, ["foo", "bar"])
writer.writerow({1:"hello", 2:"world"})
msg201732 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-30 13:47
I would argue that the TypeError is correct (field names must be strings), even though the way it is generated is a bit unorthodox :)

Let's see what others think.
msg201733 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-30 13:56
Rereading my post I disagree with myself.  ValueError is probably better in this context (the difference between ValueError and TypeError is a bit grey, and Python is not necessarily completely consistent about it.)
msg201735 - (view) Author: Tomas Grahn (tomas_grahn) Date: 2013-10-30 14:09
If non-string field names aren't allowed then shouldn't they be caught at an earlier stage, rather then when the user feeds writerow a dict with an unexpected key?

But why should the field names have to be strings in the first place? Everything else is passed through str before being written anyway...
msg201738 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-10-30 14:20
> But why should the field names have to be strings in the first place? Everything else is passed through str before being written anyway...

Good point.
msg202680 - (view) Author: Vajrasky Kok (vajrasky) * Date: 2013-11-12 08:59
Here is the preliminary patch.
msg203408 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-19 18:25
New changeset 6e5afeada7ca by R David Murray in branch '3.3':
#19449: Handle non-string keys when generating 'fieldnames' error.
http://hg.python.org/cpython/rev/6e5afeada7ca

New changeset ee2c80eeca2a by R David Murray in branch 'default':
Merge: #19449: Handle non-string keys when generating 'fieldnames' error.
http://hg.python.org/cpython/rev/ee2c80eeca2a

New changeset e52d7b173ab5 by R David Murray in branch '2.7':
#19449: Handle non-string keys when generating 'fieldnames' error.
http://hg.python.org/cpython/rev/e52d7b173ab5
msg203412 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2013-11-19 18:28
Thanks, Tomas and Vajrasky.  I tweaked the patch slightly: Thomas's fix was better, since it doesn't incur the overhead of the repr unless an an error is detected (a micro-optimization, true, but an easy one).  I also chose to only check that 'fieldnames' is mentioned in the non-variable exception text; exactly how the rest of the message text is worded is not part of the API.
History
Date User Action Args
2013-11-19 18:28:49r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg203412

stage: resolved
2013-11-19 18:25:51python-devsetnosy: + python-dev
messages: + msg203408
2013-11-12 09:00:24vajraskysetcomponents: + Library (Lib)
versions: + Python 3.3, Python 3.4, - Python 2.6
2013-11-12 08:59:42vajraskysetfiles: + fix_error_message_write_fields_not_in_fieldnames.patch
nosy: + vajrasky
messages: + msg202680

2013-10-30 14:20:53r.david.murraysetmessages: + msg201738
2013-10-30 14:09:04tomas_grahnsetmessages: + msg201735
2013-10-30 13:56:25r.david.murraysetmessages: + msg201733
2013-10-30 13:47:19r.david.murraysetnosy: + r.david.murray
messages: + msg201732
2013-10-30 10:59:32tomas_grahncreate