classification
Title: Order CSV header fields
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: holdenweb, python-dev, r.david.murray, rhettinger
Priority: normal Keywords: needs review, patch

Created on 2016-08-23 16:42 by holdenweb, last changed 2016-08-30 20:47 by holdenweb. This issue is now closed.

Files
File name Uploaded Description Edit
csv.patch holdenweb, 2016-08-23 16:46 review
csv_full.patch holdenweb, 2016-08-26 17:30 review
Messages (11)
msg273486 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-23 16:42
It's sometimes annoying that a csv.DictReader doesn't retain the field ordering given in the first line of the file. Sometimes it matters.

This patch converts the reader so that it returns an OrderedDict rather than a plain dict, thereby retaining the ordering.

All tests still pass (though I haven't yet added a test to verify that the field ordering *is* retained - didn't think it was worth it if the patch won't be added, but will happily add that test otherwise).

I have updated the documentation, but was unable in the time available to find out how to correctly reference the OrderedDict class so that it was correctly hyperlinked.
msg273487 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-23 16:46
Sorry, deleted the originally submitted (incorrect) patch file.
msg273489 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-08-23 16:52
I think this seems reasonable, now that OrderedDict is in C.
msg273530 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-24 02:12
This looks like a nice improvement.  The patch needs a test and the docs need a versionchanged entry.
msg273540 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-24 08:37
Testing could be interesting. I'm thinking of generating five random string keys with a couple of rows of data, creating csv StringIOs (using pure Python) for all 120 combinations and verifying that they read back in the order they were written.

We should also test that OrderedDicts write correctly with a DictWriter and the same key sets.

It's a fairly haphazard test plan, so I'll be happy to hear more thorough suggestions. We could at a pinch reduce the number of keys to three if speed considerations dictate.

BTW, what happened to NEWS.txt? :)
msg273694 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-26 04:16
Consider using itertools.permutations() to generate the 120 cases cases.

The news entry goes into Misc/NEWS (there is not .txt extension).
msg273695 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-26 06:05
Is there another way? :)

Sent from my iPhone

> On 26 Aug 2016, at 12:16, Raymond Hettinger <report@bugs.python.org> wrote:
> 
> 
> Raymond Hettinger added the comment:
> 
> Consider using itertools.permutations() to generate the 120 cases cases.
> 
> The news entry goes into Misc/NEWS (there is not .txt extension).
> 
> ----------
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue27842>
> _______________________________________
msg273709 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-26 17:30
OK, here's what I think should be close to the final patch. I've updated the documentation, rebuilt it and verified it reads OK, and confirmed that the new code passes all tests except those skipped for platform reasons (I think they expect a Windows environment). This includes the new test to confirm that ordering is retained over all 120 possible combinations of five keys.

Please let me know if any further updates are needed.
msg273956 - (view) Author: Roundup Robot (python-dev) Date: 2016-08-30 19:35
New changeset bb3e2a5be31b by Raymond Hettinger in branch 'default':
Issue #27842: The csv.DictReader now returns rows of type OrderedDict.
https://hg.python.org/cpython/rev/bb3e2a5be31b
msg273957 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2016-08-30 19:37
Thanks Steve.
msg273963 - (view) Author: Steve Holden (holdenweb) * (Python committer) Date: 2016-08-30 20:47
A pleasure. Pretty heavily committed at present, but all Python related so maybe there'll be more small positive improvements.
History
Date User Action Args
2016-08-30 20:47:27holdenwebsetmessages: + msg273963
2016-08-30 19:37:11rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg273957
2016-08-30 19:35:59python-devsetnosy: + python-dev
messages: + msg273956
2016-08-26 17:30:24holdenwebsetfiles: + csv_full.patch

messages: + msg273709
2016-08-26 06:05:28holdenwebsetmessages: + msg273695
2016-08-26 04:16:54rhettingersetmessages: + msg273694
2016-08-24 08:37:49holdenwebsetmessages: + msg273540
2016-08-24 02:12:24rhettingersetassignee: rhettinger

messages: + msg273530
nosy: + rhettinger
2016-08-23 16:52:32r.david.murraysettype: enhancement

messages: + msg273489
nosy: + r.david.murray
2016-08-23 16:47:53r.david.murraysethgrepos: - hgrepo353
2016-08-23 16:46:13holdenwebsetfiles: + csv.patch

messages: + msg273487
2016-08-23 16:43:34holdenwebsetfiles: - csv.patch
2016-08-23 16:42:42holdenwebcreate