classification
Title: Document order-preserving dictionary output in json
Type: enhancement Stage:
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Dima.Tisnek, aeros167, docs@python, inada.naoki, josh.r, lebigot, rhettinger
Priority: normal Keywords:

Created on 2017-06-02 09:47 by lebigot, last changed 2019-07-21 20:51 by aeros167.

Messages (8)
msg295001 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2017-06-02 09:47
The JSON encoder for dictionaries preserves the order of the items in a dictionary: it would be useful to document this behavior, so that users can rely on it.

While JSON itself does not have ordered key/value pairs, this feature can be useful (for modifying the original JSON document while maintaining as much of it untouched, for manipulating our own extension of JSON with ordered dictionaries, etc.).

The documentation could also usefully mention that _reading_ sets of key/value pairs can also be done in an order-preserving way with `object_pairs_hook=OrderedDict`.

Reference: the json module does not alter the order of the dictionary items upon encoding: https://github.com/python/cpython/blob/aacd53f6cb96fe8c4fe9ce894f22e25f356a97c3/Lib/json/encoder.py#L355.
msg347712 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2019-07-11 22:36
Is there a reason to document object_pairs_hook=OrderedDict rather than just making the decoder populate a regular dict in an order-preserving way? (No idea if it does this already)
msg347857 - (view) Author: Kyle Stanley (aeros167) * Date: 2019-07-13 21:57
> The JSON encoder for dictionaries preserves the order of the items in a dictionary

From what I can tell, the JSON encoder does nothing to preserve the order of the keys. Although the default option is to not modify the existing dictionary, the items() method returns the k,v pairs in a non-random arbitrary manner, which may not match the order the values were entered in. See https://docs.python.org/2/library/stdtypes.html#dict.items.
msg347938 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2019-07-14 20:06
Kyle, what you're saying is correct but is unfortunately unrelated to any of the points in the original issue. In fact, the JSON encoder does preserve whatever order the dictionary elements are in, and it would be useful to document this. Thus, if a user gives an OrderedDict (or a dict with a known order, in Python 3.7+), it is useful that he know that the order of its elements will not be changed upon transformation into JSON.
msg348227 - (view) Author: Kyle Stanley (aeros167) * Date: 2019-07-21 07:07
>Thus, if a user gives an OrderedDict (or a dict with a known order, in Python 3.7+), it is useful that >he know that the order of its elements will not be changed upon transformation into JSON.

I would agree that it could be helpful to document that if a user passes an OrderedDict, the order of the elements will not changed, I was just pointing out that a default dictionary implementation would not normally retain the order. 

However, after looking through the 3.7 changes and reading the discussions, it looks like dictionaries retaining insertion order was decided to be made an official part of the language. If the order matters though, it would make sense to recommend users to use OrderedDict. The order in the default dictionary implementation can end up becoming sorted if pprint is used.

Discussions: https://mail.python.org/pipermail/python-dev/2017-December/151283.html and https://mail.python.org/pipermail/python-dev/2017-December/151376.html
msg348232 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-07-21 08:52
OrderedDict is not recommended to preserve order any more.

Note that mention about `object_pairs_hook=OrderedDict` is removed intentionally.
https://github.com/python/cpython/pull/5001

Please forget about OrderedDict unless you need additional feature OrderedDict provides.  It is far inefficient than regular dict.
msg348239 - (view) Author: Eric O. LEBIGOT (lebigot) Date: 2019-07-21 12:18
The essence of the original post is simply to document any order-preserving properties of the Python to JSON and JSON to Python transformation for mappings. This way users can rely on it. This is for instance useful if a program modifies JSON created by a user and wants to preserve the user ordering.
msg348265 - (view) Author: Kyle Stanley (aeros167) * Date: 2019-07-21 20:51
> OrderedDict is not recommended to preserve order any more.

> Please forget about OrderedDict unless you need additional feature OrderedDict provides.  It is far inefficient than regular dict.

Thanks for the clarification. Should this recommendation be briefly mentioned in the documentation (https://docs.python.org/3.9/library/collections.html#ordereddict-objects)? 

Currently, the docs reference that as of 3.7, the order-preserving behavior of standard dictionaries is guaranteed, but there is no mention of the significant differences in efficiency. Based on the current description, a user might end up using OrderedDictionary on the basis that they *might* have a need for reordering elements. 

If the difference in performance is significant enough, it would be worth noting in the docs so that users can take it into consideration when deciding which data structure to use.
History
Date User Action Args
2019-07-21 20:51:38aeros167setmessages: + msg348265
2019-07-21 12:18:55lebigotsetmessages: + msg348239
2019-07-21 08:52:49inada.naokisetnosy: + inada.naoki
messages: + msg348232
2019-07-21 07:07:41aeros167setmessages: + msg348227
2019-07-14 20:06:39lebigotsetmessages: + msg347938
2019-07-13 21:57:40aeros167setnosy: + aeros167
messages: + msg347857
2019-07-11 22:36:47josh.rsetnosy: + josh.r
messages: + msg347712
2019-07-11 06:37:15xtreaksetnosy: + rhettinger

versions: + Python 3.7, Python 3.8, Python 3.9, - Python 3.6
2019-07-11 06:16:47Dima.Tisneksetnosy: + Dima.Tisnek
2017-06-02 09:49:46lebigotsetassignee: docs@python

components: + Documentation, - Library (Lib)
nosy: + docs@python
2017-06-02 09:48:20lebigotsettitle: Document order-preserving dictionary output in son -> Document order-preserving dictionary output in json
2017-06-02 09:48:14lebigotsettitle: Document order-preserving dictionary output -> Document order-preserving dictionary output in son
2017-06-02 09:47:27lebigotcreate