This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: json library is decoding/encoding when it should not
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.2, Python 3.3, Python 2.7
process
Status: closed Resolution: wont fix
Dependencies: Superseder:
Assigned To: bob.ippolito Nosy List: bob.ippolito, cvrebert, eric.araujo, ezio.melotti, flox, hac.man, petri.lehtinen, pitrou, rhettinger, thinred, vstinner
Priority: high Keywords:

Created on 2011-10-18 18:00 by thinred, last changed 2022-04-11 14:57 by admin. This issue is now closed.

Messages (21)
msg145847 - (view) Author: Tomasz Buchert (thinred) Date: 2011-10-18 18:00
Hi there!
According to http://www.ietf.org/rfc/rfc4627.txt . The JSON object is either a list or a dictionary with other elements in it. However Python's implementation took liberty to parse also basic types, like integers, strings, floats, etc. For example:

>>> import json
>>> json.loads('42')
42
>>> json.dumps(3.141592)
'3.141592'

It wouldn't be a problem if other implementations would adhere to this behavior, but this is not true. Take Ruby for example:

irb(main):007:0> require 'rubygems';
irb(main):008:0* require 'json';
irb(main):009:0* JSON.parse '42'
JSON::ParserError: 743: unexpected token at '42'
	from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in `parse'
	from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:148:in `parse'
	from (irb):9
	from :0
irb(main):010:0> JSON.generate(3.141592)
JSON::GeneratorError: only generation of JSON objects or arrays allowed
	from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:216:in `generate'
	from /var/lib/gems/1.8/gems/json-1.6.1/lib/json/common.rb:216:in `generate'
	from (irb):10
	from :0

Also http://jsonlint.com/ considers it an error.

I know this is not very important, but I've ran into this today when trying to parse JSON in Ruby which was produced in Python.

I've made a patch for that, but I still need to fix tests and there are many of them. I first would like to know if it is going to be fixed, then I will complete the patch.

Thanks for Python, anyway!

Tomek
msg145853 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-10-18 18:51
I don't see any way of changing this without breaking lots of code that currently works fine.  It may be possible to add a strict-mode (off by default) or to add warnings.
msg145888 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-19 07:05
I wonder if there are concrete use cases for loading/dumping basic types.
I see two options here:
1) we decide that load/dump should accept and produce only complete json object and go through a deprecation process, raising warnings first and errors then;
2) we decide that the correct behavior (loading/dumping basic types) is fine and that should be up to the user to check if the document has an array/object at the top level and then just document it (if it's not documented already).
msg145895 - (view) Author: Tomasz Buchert (thinred) Date: 2011-10-19 08:29
It wouldn't be a problem if 'json' library would be a thing specific to Python. But JSON is used to gule together different technologies and programming languages. I'm afraid that the default behavior should be 1).
msg145916 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-10-19 15:57
I'm thinking that we should offer a strict-option (off by default) and leave it at that.  No need to break existing code.   AFAICT, the OP is the only one to have had problems with this in-practice (the JSON module or its ancestor simplejson has been around for a long time).

Am marking this as a high priority but want to leave it open for a good while to get more comments before taking any action.
msg145917 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-19 16:05
Agreed that a strict parameter would be a good idea to implement this.
Since it's technically a new feature (and not that important), I think only 3.3 should receive it.
msg145995 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-20 00:43
I'm not sure adding a strict option is a good idea.  HTMLParser got a strict argument and it's causing more problems than it solves.  If another problem comes up later we would either have to change the behavior for strict and break code that by then is using strict=True or add yet another really_strict argument.
msg146142 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2011-10-21 22:05
> HTMLParser got a strict argument and it's causing more problems than it solves.
Funny, I used the strict mode to solve a parsing bug in a personal project.
msg146144 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-10-21 22:09
>> HTMLParser got a strict argument and it's causing more problems than it solves.
> Funny, I used the strict mode to solve a parsing bug in a personal project.
See http://bugs.python.org/issue7311#msg133075 and following messages.r
msg146147 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2011-10-21 22:56
Assigning back to me (the current maintainer of the module).
msg146152 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-10-21 23:56
> HTMLParser got a strict argument and it's causing more problems than it 
> solves.

Well, HTML is a big can of worms while JSON is a rather clean format. We can hope that adding a strict argument to json wouldn't cause such problems.
msg168941 - (view) Author: Tomasz Buchert (thinred) Date: 2012-08-23 13:54
Take a look at http://bugs.python.org/issue14674.
A proposed documentation patch specifically discusses this issue.
msg168960 - (view) Author: Petri Lehtinen (petri.lehtinen) * (Python committer) Date: 2012-08-23 19:20
The strict flag sounds good to me, too. I don't think it should be called "strict" though, as that would imply that we comply with RFC 4627 strictly (which is not true without passing allow_nan=False for dump() or  passing a parse_constant function for load()). How about "encode_any" and "decode_any" or "any_value"?

Deprecation doesn't sound good to me. If the feature is already there, it probably has its users and use cases.
msg170158 - (view) Author: Aaron (hac.man) Date: 2012-09-10 04:51
I think it's worth pointing out that both Firefox and Chrome support the non-standard JSON that Python supports (serializing and deserializing basic types).  I'm guessing that communicating with web browsers is the vast majority of JSON IPC.  That is to say, supporting the de-facto standard implemented by web browsers may be better than adhering to the exact specifications of the RFC.

Maybe someone else wants to check what IE, Safari, Opera, and the various phone browsers allow as that might influece the discussion one way or another.
msg170194 - (view) Author: Éric Araujo (eric.araujo) * (Python committer) Date: 2012-09-10 14:56
> I don't think it should be called "strict" though, as that would imply that we comply with RFC 4627 strictly
> (which is not true without passing allow_nan=False for dump() or  passing a parse_constant function for load())

What if the strict mode implied both of these things?

FWIW, I’m +1 for a doc section about how to achieve strict mode with special arguments and callbacks (if the recent doc patch does not already have that), and +0 on adding it to the code itself.
msg213991 - (view) Author: Chris Rebert (cvrebert) * Date: 2014-03-18 17:09
> FWIW, I’m +1 for a doc section about how to achieve strict mode with special arguments and callbacks (if the recent doc patch does not already have that)

The docs added by that patch do indeed cover this:
http://docs.python.org/2/library/json.html#standard-compliance
msg218642 - (view) Author: Chris Rebert (cvrebert) * Date: 2014-05-16 04:42
Note that, per the new JSON RFC 7159 (https://tools.ietf.org/html/rfc7159 ), top-level non-collection values are now de-jure permissible in JSON:

> Appendix A.  Changes from RFC 4627
>    o  Changed the definition of "JSON text" so that it can be any JSON
>       value, removing the constraint that it be an object or array.
msg219142 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2014-05-26 08:00
As Chris Rebert mentioned, the JSON standards have adopted this (unsurprising) behavior. Ruby hasn't, and I doubt Crockford has, but I think they're in the minority at this point. JavaScript's own JSON implementation works the same way json/simplejson does.

> JSON.parse(JSON.stringify('yay'))
"yay"

At best, I think it's probably worth a mention in the documentation, but not worth changing any code over.
msg219143 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2014-05-26 08:01
In other words, I would consider this to be fixed by the documentation change made elsewhere.
msg219401 - (view) Author: Chris Rebert (cvrebert) * Date: 2014-05-30 16:43
Okay, so can this issue be closed in light of the existing docs and issue 21514 then?
msg231772 - (view) Author: Chris Rebert (cvrebert) * Date: 2014-11-27 18:22
Ping!
Seems like this should be closed since the new RFC explicitly legalizes the feature in question and since the docs explicitly warn about the interoperability of the feature.
History
Date User Action Args
2022-04-11 14:57:22adminsetgithub: 57421
2014-11-27 19:34:05serhiy.storchakasetstatus: open -> closed
resolution: wont fix
stage: needs patch -> resolved
2014-11-27 18:22:25cvrebertsetmessages: + msg231772
2014-05-30 16:43:42cvrebertsetmessages: + msg219401
2014-05-26 08:01:54bob.ippolitosetmessages: + msg219143
2014-05-26 08:00:19bob.ippolitosetmessages: + msg219142
2014-05-26 07:46:41rhettingersetassignee: rhettinger -> bob.ippolito

nosy: + bob.ippolito
2014-05-16 04:42:02cvrebertsetmessages: + msg218642
2014-03-18 17:09:24cvrebertsetmessages: + msg213991
2012-09-10 14:56:35eric.araujosetmessages: + msg170194
2012-09-10 04:51:16hac.mansetnosy: + hac.man
messages: + msg170158
2012-08-24 07:36:17cvrebertsetnosy: + cvrebert
2012-08-23 19:20:20petri.lehtinensetnosy: + petri.lehtinen
messages: + msg168960
2012-08-23 13:54:49thinredsetmessages: + msg168941
2011-10-21 23:56:01pitrousetmessages: + msg146152
2011-10-21 23:47:28ezio.melottisetpriority: normal -> high
2011-10-21 22:56:57rhettingersetassignee: rhettinger
messages: + msg146147
2011-10-21 22:09:45ezio.melottisetmessages: + msg146144
2011-10-21 22:05:32eric.araujosetnosy: + eric.araujo
messages: + msg146142
2011-10-20 00:43:45ezio.melottisetpriority: high -> normal
assignee: rhettinger -> (no value)
messages: + msg145995
2011-10-19 16:05:24pitrousetnosy: + pitrou
messages: + msg145917
2011-10-19 15:58:46vstinnersetnosy: + vstinner
2011-10-19 15:57:31rhettingersetpriority: normal -> high
assignee: rhettinger
messages: + msg145916
2011-10-19 08:29:37thinredsetmessages: + msg145895
2011-10-19 07:05:17ezio.melottisetnosy: + ezio.melotti
messages: + msg145888
2011-10-18 18:51:18rhettingersetnosy: + rhettinger
messages: + msg145853
2011-10-18 18:47:49floxsetnosy: + flox
stage: needs patch
type: behavior

versions: - Python 2.6, Python 3.1, Python 3.4
2011-10-18 18:00:31thinredcreate