classification
Title: Document order-preserving dictionary output in json
Type: enhancement Stage: resolved
Components: Documentation Versions: Python 3.9, Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Dima.Tisnek, aeros167, docs@python, inada.naoki, josh.r, lebigot, rhettinger
Priority: low Keywords: patch

Created on 2017-06-02 09:47 by lebigot, last changed 2019-08-22 22:51 by rhettinger. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 15397 merged rhettinger, 2019-08-22 17:29
PR 15403 merged miss-islington, 2019-08-22 22:15
Messages (12)
msg295001 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2017-06-02 09:47
The JSON encoder for dictionaries preserves the order of the items in a dictionary: it would be useful to document this behavior, so that users can rely on it.

While JSON itself does not have ordered key/value pairs, this feature can be useful (for modifying the original JSON document while maintaining as much of it untouched, for manipulating our own extension of JSON with ordered dictionaries, etc.).

The documentation could also usefully mention that _reading_ sets of key/value pairs can also be done in an order-preserving way with `object_pairs_hook=OrderedDict`.

Reference: the json module does not alter the order of the dictionary items upon encoding: https://github.com/python/cpython/blob/aacd53f6cb96fe8c4fe9ce894f22e25f356a97c3/Lib/json/encoder.py#L355.
msg347712 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-07-11 22:36
Is there a reason to document object_pairs_hook=OrderedDict rather than just making the decoder populate a regular dict in an order-preserving way? (No idea if it does this already)
msg347857 - (view) Author: Kyle Stanley (aeros167) * (Python triager) Date: 2019-07-13 21:57
> The JSON encoder for dictionaries preserves the order of the items in a dictionary

From what I can tell, the JSON encoder does nothing to preserve the order of the keys. Although the default option is to not modify the existing dictionary, the items() method returns the k,v pairs in a non-random arbitrary manner, which may not match the order the values were entered in. See https://docs.python.org/2/library/stdtypes.html#dict.items.
msg347938 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2019-07-14 20:06
Kyle, what you're saying is correct but is unfortunately unrelated to any of the points in the original issue. In fact, the JSON encoder does preserve whatever order the dictionary elements are in, and it would be useful to document this. Thus, if a user gives an OrderedDict (or a dict with a known order, in Python 3.7+), it is useful that he know that the order of its elements will not be changed upon transformation into JSON.
msg348227 - (view) Author: Kyle Stanley (aeros167) * (Python triager) Date: 2019-07-21 07:07
>Thus, if a user gives an OrderedDict (or a dict with a known order, in Python 3.7+), it is useful that >he know that the order of its elements will not be changed upon transformation into JSON.

I would agree that it could be helpful to document that if a user passes an OrderedDict, the order of the elements will not changed, I was just pointing out that a default dictionary implementation would not normally retain the order. 

However, after looking through the 3.7 changes and reading the discussions, it looks like dictionaries retaining insertion order was decided to be made an official part of the language. If the order matters though, it would make sense to recommend users to use OrderedDict. The order in the default dictionary implementation can end up becoming sorted if pprint is used.

Discussions: https://mail.python.org/pipermail/python-dev/2017-December/151283.html and https://mail.python.org/pipermail/python-dev/2017-December/151376.html
msg348232 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-07-21 08:52
OrderedDict is not recommended to preserve order any more.

Note that mention about `object_pairs_hook=OrderedDict` is removed intentionally.
https://github.com/python/cpython/pull/5001

Please forget about OrderedDict unless you need additional feature OrderedDict provides.  It is far inefficient than regular dict.
msg348239 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2019-07-21 12:18
The essence of the original post is simply to document any order-preserving properties of the Python to JSON and JSON to Python transformation for mappings. This way users can rely on it. This is for instance useful if a program modifies JSON created by a user and wants to preserve the user ordering.
msg348265 - (view) Author: Kyle Stanley (aeros167) * (Python triager) Date: 2019-07-21 20:51
> OrderedDict is not recommended to preserve order any more.

> Please forget about OrderedDict unless you need additional feature OrderedDict provides.  It is far inefficient than regular dict.

Thanks for the clarification. Should this recommendation be briefly mentioned in the documentation (https://docs.python.org/3.9/library/collections.html#ordereddict-objects)? 

Currently, the docs reference that as of 3.7, the order-preserving behavior of standard dictionaries is guaranteed, but there is no mention of the significant differences in efficiency. Based on the current description, a user might end up using OrderedDictionary on the basis that they *might* have a need for reordering elements. 

If the difference in performance is significant enough, it would be worth noting in the docs so that users can take it into consideration when deciding which data structure to use.
msg348275 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-07-22 01:09
Kyle, the link you gave already discusses performance to the extent that it matters.  Please keep focused on the OP's thought that the JSON docs should explicitly promised that key/value pairs are added in the order that they are encountered.  The existing JSON docs imply this pretty strongly, but it can be made more explicit.
msg348276 - (view) Author: Kyle Stanley (aeros167) * (Python triager) Date: 2019-07-22 02:06
> Please keep focused on the OP's thought that the JSON docs should explicitly promised that
> key/value pairs are added in the order that they are encountered. The existing JSON docs imply
> this pretty strongly, but it can be made more explicit.

I agree that the behavior could be more explicitly stated in the docs, as that could be potentially useful information for users to be aware of. 

Apologies for derailing the topic, I became too focused on the OrderedDict vs Dict topic, which was only a small part of the issue. Any further discussion on that topic or updating the OrderedDict docs should probably be moved into it's own issue, if at all.
msg350234 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-08-22 22:14
New changeset 657008ea0336ff4f275ed3f0c2b6dd2e52de2bba by Raymond Hettinger in branch 'master':
bpo-30550: Clarify JSON ordering guarantees (GH-15397)
https://github.com/python/cpython/commit/657008ea0336ff4f275ed3f0c2b6dd2e52de2bba
msg350237 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2019-08-22 22:50
New changeset 4a40498ea96a3c606952712c7951b2ea4ab258e4 by Raymond Hettinger (Miss Islington (bot)) in branch '3.8':
bpo-30550: Clarify JSON ordering guarantees (GH-15397) (GH-15403)
https://github.com/python/cpython/commit/4a40498ea96a3c606952712c7951b2ea4ab258e4
History
Date User Action Args
2019-08-22 22:51:19rhettingersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2019-08-22 22:50:37rhettingersetmessages: + msg350237
2019-08-22 22:15:49miss-islingtonsetpull_requests: + pull_request15110
2019-08-22 22:14:47rhettingersetmessages: + msg350234
2019-08-22 17:29:28rhettingersetkeywords: + patch
stage: patch review
pull_requests: + pull_request15105
2019-07-22 02:06:39aeros167setmessages: + msg348276
2019-07-22 01:10:11rhettingersetpriority: normal -> low
versions: - Python 3.7
2019-07-22 01:09:34rhettingersetmessages: + msg348275
2019-07-22 00:26:44rhettingersetassignee: docs@python -> rhettinger
2019-07-21 20:51:38aeros167setmessages: + msg348265
2019-07-21 12:18:55lebigotsetmessages: + msg348239
2019-07-21 08:52:49inada.naokisetnosy: + inada.naoki
messages: + msg348232
2019-07-21 07:07:41aeros167setmessages: + msg348227
2019-07-14 20:06:39lebigotsetmessages: + msg347938
2019-07-13 21:57:40aeros167setnosy: + aeros167
messages: + msg347857
2019-07-11 22:36:47josh.rsetnosy: + josh.r
messages: + msg347712
2019-07-11 06:37:15xtreaksetnosy: + rhettinger

versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.6
2019-07-11 06:16:47Dima.Tisneksetnosy: + Dima.Tisnek
2017-06-02 09:49:46lebigotsetassignee: docs@python

components: + Documentation, - Library (Lib)
nosy: + docs@python
2017-06-02 09:48:20lebigotsettitle: Document order-preserving dictionary output in son -> Document order-preserving dictionary output in json
2017-06-02 09:48:14lebigotsettitle: Document order-preserving dictionary output -> Document order-preserving dictionary output in son
2017-06-02 09:47:27lebigotcreate