classification
Title: json encoder unable to handle decimal
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, bob.ippolito, christian.heimes, cvrebert, eric.araujo, ezio.melotti, jcea, mark.dickinson, mjensen, pitrou, ralhei, rhettinger, risa2000, serhiy.storchaka, zzzeek
Priority: normal Keywords: patch

Created on 2012-11-22 22:45 by eric.araujo, last changed 2020-12-21 21:50 by rhettinger.

Files
File name Uploaded Description Edit
json_decimal.patch ralhei, 2013-07-07 08:10 Patch file generated via "hg diff" review
Messages (16)
msg176135 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-11-22 22:45
In 2.7 and other versions, the json module has incomplete support for decimals:

>>> json.loads('0.2', parse_float=Decimal)
Decimal('0.2')
>>> json.dumps(json.loads('0.2', parse_float=Decimal))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Decimal('0.2') is not JSON serializable

simplejson encodes decimals out of the box, but json can’t round-trip.
msg176136 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-11-22 22:47
See lengthy discussion that lead to inclusion in simplejson here: http://code.google.com/p/simplejson/issues/detail?id=34
msg176158 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2012-11-23 09:32
The json module already has too many options. No need for yet one such specialized.

>>> class number_str(float):
...     def __init__(self, o):
...         self.o = o
...     def __repr__(self):
...         return str(self.o)
... 
>>> def decimal_serializer(o):
...     if isinstance(o, decimal.Decimal):
...         return number_str(o)
...     raise TypeError(repr(o) + " is not JSON serializable")
... 
>>> print(json.dumps([decimal.Decimal('0.20000000000000001')], default=decimal_serializer))
[0.20000000000000001]

You can extend this to support complex numbers, fractions, date and time, and many other custom types. Have specialized options for this would be cumbersome.
msg176161 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-11-23 10:13
Judging by the discussion that Éric points to, and by the various stackoverflow questions on the topic ([1], [2]), this is a common enough need that I think it would make sense to have some support for it in the std. lib.

There's a sense in which Decimal is the 'right' type for json, and we shouldn't make it harder for people to do the right thing with respect to (e.g.) financial data in databases.





[1] http://stackoverflow.com/questions/4019856/decimal-to-json
[2] http://stackoverflow.com/questions/1960516/python-json-serialize-a-decimal-object
msg176398 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-11-26 05:09
Thanks for the workaround Serhiy, I stole that and ran with it :)

For 3.4 I still think something built-in would be best.
msg179869 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-01-13 12:29
Decimal numbers should simply be serializable by default. It doesn't make sense to add a specialized option.
msg192520 - (view) Author: Ralph Heinkel (ralhei) Date: 2013-07-07 08:10
This patch was implemented on Europython 2013 sprint. It's my first addition to Python core ever so please bear with me if it's not perfect. 

Decimal support is implemented both in the C and Python JSON code.

There is one peculiarity to mention about the Decimal addition in function _json.c:encoder_listencode_obj() of my patch:
The addition of

    else if (PyObject_IsInstance(obj, (PyObject*)PyDecimalType)) {
        PyObject *encoded = encoder_encode_decimal(s, obj);
        if (encoded == NULL)
            return -1;
        return _steal_accumulate(acc, encoded);
    }

was has to be located AFTER lists and dicts are handled in the JSON encoder, otherwise the unittest "test_highly_nested_objects_encoding()" from test_recursion.py fails with a nasty, unrecoverable Python exception.
My guess is that this is due additional stack allocation when the stack space is almost used up by the deeply nested recursion code.
msg213381 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2014-03-13 10:30
Patch looks good and contains tests for the C and Python code.

Documentation is missing (a note to tell that json.dump converts decimal.Decimal instances to JSON numbers, a versionchanged directive, maybe a link to the doc that explains parse_float=decimal.Decimal).
msg214733 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-24 21:31
The patch isn't really ok, IMO. It forcibly imports the decimal module and then looks up the type there. The decimal module is a rather large one and it shouldn't get imported if it doesn't get used.

I think it would be better to rely on the __float__ special method, which would also automatically accept other numberish types such as Fraction.
msg214737 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-03-24 21:38
I think we should really apply #19232. At least that would take care
of the import issue.
msg224032 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2014-07-26 09:13
I'm EuroPython 2014 in Berlin. Ralph has approached me and asked me about progress on the progress of this patch. I'm reluctant to implement a special case for decimals for two reasons:

1) JSON just support floats and decimals are IMHO incompatible with floats. The conversion of decial to JSON floats is a loosely operation.

2) Rather than having a special case I would rather go with a general implementation that uses an ABC to JSON dump some float-like objects.
msg224096 - (view) Author: Chris Rebert (cvrebert) * Date: 2014-07-27 02:17
> 1) JSON just support floats

If you read the JSON standards documents, you'll see that this isn't accurate.

Regardless, a general solution for non-built-in numeric types does seem preferable.
msg274026 - (view) Author: Mads Jensen (mjensen) Date: 2016-08-31 15:02
Hi @cvrebert and team - do you know if this was ever implemented. It seems that it is still an issue for financial applications, and that the solution proposed would be relevant and helpful.
msg289010 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-03-05 13:39
The trick from msg176158 no longer works since issue26719.
msg349250 - (view) Author: Richard Musil (risa2000) Date: 2019-08-08 19:46
It looks like I am resurrecting an old item, but I have been just hit by this and was directed to this issue (https://mail.python.org/archives/list/python-ideas@python.org/thread/WT6Z6YJDEZXKQ6OQLGAPB3OZ4OHCTPDU/)

I wonder if adding something similar to what `simplejson` uses (i.e. explicitly specifying in `json.dump(s)` how to serialize `decimal.Decimal`) could be acceptable.

Or, the other idea would be to expose a method in JSONEncoder, which would accept "raw" textual output, i.e. string (or even `bytes`) and would encode it without adding additional characters to it. (as explained in my posts in the other threads).

As it seems right now, there is no way to serialize `decimal.Decimal` the same way it is deserialized, i.e. while preserving the (arbitrary) precision.
msg383559 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-12-21 21:50
> I wonder if adding something similar to what `simplejson` uses 
> (i.e. explicitly specifying in `json.dump(s)` how to
> serialize `decimal.Decimal`) could be acceptable.

+1 for this approach.  For financial applications, we need the recommended solution to be simple.
History
Date User Action Args
2020-12-21 21:50:19rhettingersetnosy: + bob.ippolito

messages: + msg383559
versions: + Python 3.10, - Python 3.5
2020-12-20 23:45:32zzzeeksetnosy: + zzzeek
2019-08-08 19:46:27risa2000setnosy: + risa2000
messages: + msg349250
2017-03-05 13:39:41serhiy.storchakasetmessages: + msg289010
2016-08-31 15:02:20mjensensetnosy: + mjensen
messages: + msg274026
2014-10-14 17:01:36skrahsetnosy: - skrah
2014-07-27 02:17:06cvrebertsetmessages: + msg224096
2014-07-26 09:13:05christian.heimessetmessages: + msg224032
2014-07-26 08:08:02ralheisetnosy: + christian.heimes
2014-03-24 21:38:57skrahsetmessages: + msg214737
2014-03-24 21:31:55pitrousetmessages: + msg214733
2014-03-24 20:57:58cvrebertsetnosy: + cvrebert
2014-03-13 10:30:46eric.araujosetstage: needs patch -> patch review
messages: + msg213381
versions: + Python 3.5, - Python 3.4
2014-03-13 10:23:28eric.araujosethgrepos: - hgrepo202
2013-07-07 08:10:03ralheisetfiles: + json_decimal.patch

nosy: + ralhei
messages: + msg192520

hgrepos: + hgrepo202
keywords: + patch
2013-01-13 17:56:59Arfreversetnosy: + Arfrever
2013-01-13 12:29:42pitrousettype: behavior -> enhancement
messages: + msg179869
components: + Library (Lib)
2012-11-30 16:53:32jceasetnosy: + jcea
2012-11-26 05:09:07eric.araujosetmessages: + msg176398
2012-11-23 10:13:05mark.dickinsonsetversions: + Python 3.4
nosy: + mark.dickinson

messages: + msg176161

stage: needs patch
2012-11-23 09:32:56serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg176158
2012-11-22 23:07:29skrahsetnosy: + skrah
2012-11-22 22:47:48eric.araujosetmessages: + msg176136
2012-11-22 22:45:38eric.araujocreate