This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: MappingProxy objects should JSON serialize just like a dictionary
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: bob.ippolito Nosy List: Michael Smith2, Nathaniel Manista, bob.ippolito, eric.araujo, rhettinger, serhiy.storchaka, zgoda.mobica
Priority: normal Keywords:

Created on 2018-10-01 01:56 by Michael Smith2, last changed 2022-04-11 14:59 by admin.

Messages (10)
msg326756 - (view) Author: Michael Smith (Michael Smith2) Date: 2018-10-01 01:56
If a mappingproxy object is a read-only proxy to a mapping, would it make sense for them to JSON serialize just like the mapping they come from? Currently, json.dumps throws the "I don't know how to serialize this" error:

$ python -c 'import json
> import types
> json.dumps(types.MappingProxyType({}))'
Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/lib64/python3.6/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib64/python3.6/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib64/python3.6/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib64/python3.6/json/encoder.py", line 180, in default
    o.__class__.__name__)
TypeError: Object of type 'mappingproxy' is not JSON serializable
msg326761 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-10-01 03:26
IIRC, there was an early decision to handle only exact types and their subclasses (plus tuples).  For example, os.environ and instances of collections.ChainMap are not JSON serializable.

I believe the reason was that it made encoding faster, more predictable, and more likely to match round-trip expectations.  For those wanting more generality, there are at least two options.  The simplest option is to coerce the input to supported type.  A more complete solution is to write a subclass of JSONEncoder to pass into json.dump() as the *cls* argument (there are examples of how to do this in the docs).

For the specific case of mappingproxy, there is another issue.  Multiple components of a class dict are not all JSON serializable, so you have the same problem yet again with getset_descriptor objects, member objects, and various slot wrappers.
msg326845 - (view) Author: Michael Smith (Michael Smith2) Date: 2018-10-02 03:01
OK, I appreciate the response. The MappingProxy objects I am working with are in a nested data structure, so accessing them to coerce them directly isn't feasible. I created 

class MappingProxyEncoder(JSONEncoder):
  def default(self, obj):
    if isinstance(obj, MappingProxyType):
      return obj.copy()
    return JSONEncoder.default(self, obj)

and using that works fine for json.dumps (passes all my tests, at least). But I suspect that obj.copy() is more expensive than the code I'm replacing, which is a local implementation of ImmutableDict I was trying to do away with, following a readthrough of PEP 416. I wonder if there's a cheaper way to get at the underlying dictionary.

I suspect I'm not the only python user who foolishly expected MappingProxy to be a drop-in replacement for local ImmutableDict implementations. Maybe this ticket will be useful for others to read.
msg326849 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-10-02 04:25
Using the default method or argument is a proper way of serializing custom types to JSON. There are a number of issues fr adding support for serializing different types by default. I work on a patch that will allow to make different types serializable in a more convenient way.
msg343617 - (view) Author: (zgoda.mobica) Date: 2019-05-27 12:42
In particular case of lazy type proxy objects the result of dump() may be different than dumps() because dump does iterencode() over data and possibly causing proxy object evaluation. In such case dumps() will raise TypeError but dump() will not.
msg370826 - (view) Author: Nathaniel Manista (Nathaniel Manista) Date: 2020-06-06 15:14
There's a great deal more additional discussion about this over at https://discuss.python.org/t/json-does-not-support-mapping-and-mutablemapping/2829, but I would like to heartily "hear hear" this issue for the way it is weird that

json.dumps(frozendict.frozendict({'3':5}))

does not evaluate to the string "{'3': 5}" and it's something that makes writing according to and teaching others to code according to the make-your-code-safer-by-using-immutability-in-all-possible-things principle.
msg370828 - (view) Author: Nathaniel Manista (Nathaniel Manista) Date: 2020-06-06 15:30
Err, "... it's something that complicates writing code according to...", sorry.
msg370830 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-06-06 16:09
+1 for reconsidering whether the json module should support mappings that don't inherit for dict.  The decision not to support general mappings is ancient and predates a lot of modern tooling.

The only worry is that the objects won't round-trip — you could store a ChainMap but you wouldn't get a ChainMap back :-(
msg370844 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2020-06-06 19:01
I would certainly reconsider it at this point, I think a bona fide ABC *specific to JSON encoding* would be a good way to do it. simplejson has two ways to do this, the `for_json` parameter which will use a `for_json()` method on any object as the JSON representation if present, and the `namedtuple_as_object` parameter which will do the same for objects with an `_asdict()` method. As part of the standard library it might make more sense to rename `for_json()` to `__json__()` or similar.

It is a bit unfortunate that you can't guarantee round-trip on deserialization, but that has always been the case. To get round-tripping (without tagging everything that has been encoded in some way like pickle does), you really should be working with a schema of some kind.
msg379041 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2020-10-19 22:24
See also #30343
History
Date User Action Args
2022-04-11 14:59:06adminsetgithub: 79039
2020-10-19 22:24:41eric.araujosetnosy: + eric.araujo
messages: + msg379041
2020-06-06 19:01:26bob.ippolitosetmessages: + msg370844
2020-06-06 16:09:03rhettingersetmessages: + msg370830
2020-06-06 15:30:59Nathaniel Manistasetmessages: + msg370828
2020-06-06 15:14:16Nathaniel Manistasetnosy: + Nathaniel Manista
messages: + msg370826
2019-05-27 12:42:56zgoda.mobicasetnosy: + zgoda.mobica
messages: + msg343617
2018-10-02 04:25:10serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg326849
2018-10-02 03:01:36Michael Smith2setmessages: + msg326845
2018-10-01 03:26:23rhettingersetversions: - Python 3.4, Python 3.5, Python 3.6, Python 3.7
nosy: + rhettinger, bob.ippolito

messages: + msg326761

assignee: bob.ippolito
2018-10-01 01:56:32Michael Smith2create