classification
Title: json.dumps doesn't respect OrderedDict's iteration order
Type: Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: benjamin.peterson, lehmannro, rhettinger, wangchun
Priority: high Keywords:

Created on 2009-05-25 09:22 by wangchun, last changed 2010-10-30 07:30 by rhettinger. This issue is now closed.

Messages (9)
msg88311 - (view) Author: Wang Chun (wangchun) Date: 2009-05-25 09:22
PEP-0372 and Issue 5381 both say json.dumps respect OrderedDict's 
iteration order, but the example in them do not work on my latest trunk 
build.

$ uname -a
Linux 12.38 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008 x86_64 
x86_64 x86_64 GNU/Linux
$ python2.7
Python 2.7a0 (trunk, May 21 2009, 08:00:00) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import json
>>> from collections import OrderedDict
>>> items = [('one', 1), ('two', 2), ('three', 3), ('four', 4), ('five', 
5)]
>>> json.dumps(OrderedDict(items))
'{"four": 4, "three": 3, "five": 5, "two": 2, "one": 1}'
msg88335 - (view) Author: Robert Lehmann (lehmannro) * Date: 2009-05-26 07:00
This only seems to be the case with the C implementation of json (_json). 

>>> json.encoder.c_make_encoder = None
>>> json.dumps(OrderedDict(items))
'{"one": 1, "two": 2, "three": 3, "four": 4, "five": 5}'

I think the culprit is encoder_listencode_dict (Modules/_json.c:2049).
It uses PyDict_Next to fetch all items from the dictionary and thereby
loses order. A special code branch for dict subclasses
(!PyDict_CheckExact) which uses PyIter* instead should solve the
problem. (PyDict_Next should not honor order no matter what IMO.)

If nobody beats me to it I can try to come up with a patch.
msg88361 - (view) Author: Skip Montanaro (skip.montanaro) * (Python committer) Date: 2009-05-26 12:37
I can't see that the order of keys should matter for language-neutral
serialization libs like json or xmlrpclib.  You're quite possibly
going to be communicating with something on the other end which doesn't
have an OrderedDict-like class.  Why add the extra complication to the
library?

Even with Python of recent enough vintage at both ends, I can't see
that you could dumps() an OrderedDict then loads() the resulting
string and get an OrderedDict out as a result.  You should get a dict,
right?
msg88396 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-05-26 21:40
Skip, the json API for 2.7/3.1 includes an object_pairs_hook that allows
an OrderedDict to be generated from a json file.  The OP is pointing out
that the latest C upgrade prevents round-tripping while preserving
order.  Use cases for that were discussed in the thread for the
object_pairs_hook.  While you can't count on order preservation on both
ends, there are tools such as PHP that do preserve order.  Also, when
working with manually type user inputs (such as config files), it is
nice when updates generally keep the file in its original order so that
it is clear what had changed.
msg88405 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2009-05-27 06:52
r72961 Fixed for Py3.1.  Leaving open until backported to 2.7.
msg108411 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-06-22 19:25
Benjamin, it's a little late for this one for 2.7.  Do you prefer to close it forever or to treat it as a bug fix for 2.7.1?
msg113213 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-08-07 22:33
I think we should close it.
msg119938 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2010-10-29 21:34
Upon further consideration, I think this could be backported.
msg119964 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2010-10-30 07:30
Backported to 2.7
See r85966
History
Date User Action Args
2010-10-30 07:30:32rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg119964
2010-10-30 00:01:47rhettingersetpriority: normal -> high
2010-10-29 21:34:38benjamin.petersonsetstatus: closed -> open
assignee: benjamin.peterson -> rhettinger
resolution: wont fix -> (no value)
messages: + msg119938
2010-08-07 22:33:17benjamin.petersonsetstatus: open -> closed
resolution: wont fix
messages: + msg113213
2010-06-22 19:25:02rhettingersetassignee: rhettinger -> benjamin.peterson

messages: + msg108411
nosy: + benjamin.peterson
2010-05-20 20:32:08skip.montanarosetnosy: - skip.montanaro
2009-09-22 02:24:52rhettingersetversions: - Python 3.1
2009-05-27 06:52:59rhettingersetpriority: critical -> normal

messages: + msg88405
versions: + Python 3.1
2009-05-26 21:40:26rhettingersetmessages: + msg88396
2009-05-26 12:37:59skip.montanarosetnosy: + skip.montanaro
messages: + msg88361
2009-05-26 07:00:42lehmannrosetnosy: + lehmannro
messages: + msg88335
2009-05-26 05:26:14rhettingersetpriority: critical
2009-05-25 12:47:54benjamin.petersonsetassignee: rhettinger

nosy: + rhettinger
2009-05-25 09:22:44wangchuncreate