classification
Title: Log type of unserializable value when raising JSON TypeError
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Madison May, bob.ippolito, ezio.melotti, pitrou, rhettinger, serhiy.storchaka, terry.reedy
Priority: normal Keywords: patch

Created on 2015-07-15 20:49 by Madison May, last changed 2017-11-10 20:32 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 4364 open serhiy.storchaka, 2017-11-10 20:09
Messages (6)
msg246776 - (view) Author: Madison May (Madison May) Date: 2015-07-15 20:49
Currently the json lib only logs the string representation of the variable, which does not always include type information.

I recently ran into a difficult to debug issue with code similar to the following:

```
import json
import numpy as np
d = {'data': np.int16(5)}
json.dumps(d)
```

which produces the following error:

```
TypeError: 5 is not JSON serializable
```

It took us quite a while to determine that `5` was actually of type `np.int` instead of the native type.

A cursory glance at StackOverflow suggests that I'm not alone in running into this issue: http://stackoverflow.com/questions/10872604/json-dump-throwing-typeerror-is-not-json-serializable-on-seemingly-vali


I'd like to consider modifying the error message to be more similar to the following:

```
TypeError: 5 of type `numpy.int16` is not JSON serializable
```
msg246862 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-07-17 18:03
A typical TypeError message use the following pattern:
  TypeError: 'int' object is not callable
as it is the class, not the value, that is the problem.

If the same is always true for the JSON TypeError, at the point of failure, then the dumps message could follow the same pattern.

However, I don't know if the message *is* value independent.  The SO question asked about

TypeError: {'album': [u"Rooney's Lost Album"], 'title': [u'The Kids
After Sunset'], 'artist': [u'Rooney']} is not JSON serializable

and the OP claimed in a comment to the accepted answer that the problem was that the value objects represented as lists were not list() objects, just as '5' here is not an int() object.

The JSON error code, in 3.5 at

  File "C:\Programs\Python35\lib\json\encoder.py", line 180, in default
    raise TypeError(repr(o) + " is not JSON serializable")

could be expanded to check whether the string representation starts with the class name and if it does not, add "'classname' object" at the front. This would solve Madison's posted issue, but not the SO problem.
(It would, however, have made it clear that the {}s represented a real dict() object and directed attention inward.)

For testing, compile('','','exec') returns an object that cannot be dumped.

To me, this looks more like an enhancement than bugfix, so a change might be limited to future releases.
msg246865 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-07-17 19:07
I think it would be better to change error message to mention the type only. Yet one argument is that the repr of affected object can be very large, while the type name usually is short enough. repr() even can raise an exception (e.g. MemoryError).
msg246919 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2015-07-18 22:22
This seems like a very reasonable proposal. It would be great if we could also include a path in the error message (e.g. `obj["foo"][1]["bar"]`) as well to provide enough context to track down the error quickly.
msg306045 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-10 19:41
This issue was partially fixed in issue26623. The error message for unsupported types now is `"Object of type '%s' is not JSON serializable" % o.__class__.__name__`.

But this change is not complete. The error message for non-string keys still contains the repr of a key. And the example for default() in the module docstring contains the repr of an object.
msg306046 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-10 20:32
The idea about including a path to non-serializable object looks interesting (and it would be even more useful for pickling), but harder to implement.
History
Date User Action Args
2017-11-10 20:32:08serhiy.storchakasetmessages: + msg306046
2017-11-10 20:09:39serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request4317
2017-11-10 19:41:51serhiy.storchakasetassignee: serhiy.storchaka
stage: test needed -> (no value)
messages: + msg306045
versions: + Python 3.7, - Python 2.7, Python 3.5, Python 3.6
2017-11-10 19:33:35serhiy.storchakaunlinkissue31994 superseder
2017-11-10 19:28:24serhiy.storchakalinkissue31994 superseder
2015-07-18 22:22:13bob.ippolitosetmessages: + msg246919
2015-07-17 19:07:32serhiy.storchakasetnosy: + bob.ippolito, pitrou, serhiy.storchaka
messages: + msg246865
2015-07-17 18:03:36terry.reedysetversions: + Python 3.5, Python 3.6
nosy: + rhettinger, ezio.melotti, terry.reedy

messages: + msg246862

type: enhancement
stage: test needed
2015-07-15 20:49:32Madison Maycreate