json.loads() accepts strings but errors on bytes objects. Documentation and API indicate that both should work. Review of json/__init__.py code shows that the loads() function's 'encoding' arg is ignored and no decoding takes place before the object is passed to JSONDecoder.decode()

Tested on Python 3.1.2 and Python 3.2rc1; fails on both. 

Example:

#################################################

#!/usr/local/bin/python3.2

import json

print(json.loads('123'))
# 123

print(json.loads(b'123'))
# /Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/json/decoder.py:325:  
#   TypeError: can't use a string pattern on a bytes-like object

print(json.loads(b'123', encoding='utf-8'))
# /Library/Frameworks/Python.framework/Versions/3.1/lib/python3.1/json/decoder.py:325:  
#   TypeError: can't use a string pattern on a bytes-like object

#################################################

Patch attached.

Hmm.  According to issue 4136, all bytes support was supposed to have been removed.

Indeed, the documentation (and function docstring) needs fixing instead. It's a pity we didn't remove the useless `encoding` parameter.

Georg: Is it still time to deprecate the encoding parameter in 3.2?

I've committed a doc fix in r88137.

Doc fix works for me.

Works for me, py2.7 on snow leopard.

anthony: this is python3-only problem.

Now it's too late for 3.2, should this be done for 3.3?

If you’re talking about deprecating the obsolete encoding argument (maybe it’s time for a new bug report), +1.

I'll just mention that the elimination of bytes handling is a bit unfortunate, since this idiom which works in Python 2 no longer works:

fp = urlopen(url)
json_data = json.load(fp)

/me sad

> I'll just mention that the elimination of bytes handling is a bit
> unfortunate, since this idiom which works in Python 2 no longer works:
> 
> fp = urlopen(url)
> json_data = json.load(fp)

What if the returned JSON uses a charset other than utf-8 ?

I know this does not fix anything at the core, but it would allow you to use json.loads() with python 3.2 (maybe 3.1?):

Replace 
json.loads(raw_data)

by

raw_data = raw_data.decode('utf-8') # Or any other ISO format
json.loads(raw_data)

> What if the returned JSON uses a charset other than utf-8 ?

According to RFC 4627: "JSON text SHALL be encoded in Unicode.  The default encoding is UTF-8." RFC 4627 also offers a way to autodetect other Unicode encodings.

Well, adding support for bytes objects using the spec from RFC 4627 (or at least with utf-8 as a default) may be an enhancement for 3.3.

Things are a little more complicated. '123' is not a valid JSON according to RFC 4627 (the top-level element can only be an object or an array). This means that the autodetection algorithm will not always work for such non-standard data.

If we can parse binary data, then there must be a way to generate binary data in at least one of the Unicode encodings.

By the way, the documentation should give a link to RFC 4627 and explain the current implementation is different from it.

> Things are a little more complicated. '123' is not a valid JSON
> according to RFC 4627 (the top-level element can only be an object or
> an array). This means that the autodetection algorithm will not always
> work for such non-standard data.

The autodetection algorithm needn't examine all 4 first bytes. If the 2
first bytes are non-zero, you have UTF-8 data. Otherwise, the JSON text
will be at least 4 bytes long (since it's either UTF-16 or UTF-32).

I mean a string that starts with '\u0000'. b'"\x00...'.

Le jeudi 26 avril 2012 à 15:48 +0000, Serhiy Storchaka a écrit :
> 
> I mean a string that starts with '\u0000'. b'"\x00...'.

According to the RFC, that should be escaped:

   All Unicode characters may be placed within the
   quotation marks except for the characters that must be escaped:
   quotation mark, reverse solidus, and the control characters (U+0000
   through U+001F).

And indeed:

>>> json.loads('"\u0000"')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/opt/lib/python3.2/json/__init__.py", line 307, in loads
    return _default_decoder.decode(s)
  File "/home/antoine/opt/lib/python3.2/json/decoder.py", line 351, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/antoine/opt/lib/python3.2/json/decoder.py", line 367, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Invalid control character at: line 1 column 1 (char 1)
>>> json.loads('"\\u0000"')
'\x00'

According to current implementation this is acceptable.

>>> json.loads('"\u0000"', strict=False)
'\x00'

> According to current implementation this is acceptable.

Then perhaps auto-detection can be restricted to strict mode? Non-strict mode would always use utf-8.
Or we can just skip auto-detection altogether (I don't think many people produce utf-16 or utf-32 JSON; that would be a waste of bandwidth for no obvious benefit).

Related to this question is a question about errors. How to inform the user, if an error occurred in the decoding with detected encoding? Leave UnicodeDecodeError or convert it to ValueError? If there is a syntax error in JSON -- exception will refer to the position in the decoded string, we should to translate it to the position in the original binary string?