Inspired by simplejson issue [1] which is related to standard json module too. JSON parser 3.3+ and wide builds of 3.2- raise an error on invalid strings (i.e. with unpaired surrogate), while narrow builds and some third-party parsers. Wide builds are right, such JSON data is invalid. However it will be good to be optionally more permissive to input data. Otherwise it is not easy process such invalid data.

I propose to add an "error" parameter to JSON decoder and encoder with the same meaning as in string decoding/encoding. "strict" is default and "surrogatepass" corresponds to narrow builds (and non-strict third-party parsers).

[1] https://github.com/simplejson/simplejson/issues/62

I wonder if json should simply be less strict by default. If you pass the raw unescaped character, the json module accepts it:

>>> json.loads('{"a": "\ud8e9"}')
{'a': '\ud8e9'}

It's only if you pass the escaped representation that json rejects it:

>>> json.loads('{"a": "\\ud8e9"}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/antoine/cpython/default/Lib/json/__init__.py", line 316, in loads
    return _default_decoder.decode(s)
  File "/home/antoine/cpython/default/Lib/json/decoder.py", line 344, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/antoine/cpython/default/Lib/json/decoder.py", line 360, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Unpaired high surrogate: line 1 column 9 (char 8)

See also #11489.

After investigating the problem deeper, I see that new parameter is not needed. RFC 4627 does not make exceptions for the range 0xD800-0xDFFF, and the decoder must accept lone surrogates, both escaped and unescaped. Non-BMP characters may be represented as escaped surrogate pair, so escaped surrogate pair may be decoded as non-BMP character, while unescaped surrogate pair shouldn't.

Here is a patch, with which JSON decoder accepts encoded lone surrogates. Also fixed a bug when Python implementation decodes "\\ud834\\u0079x" as "\U0001d179".

The patch that I wrote for simplejson is here (it differs a bit from serhiy's patch): https://github.com/simplejson/simplejson/commit/35816bfe2d0ddeb5ddcc68239683cbb35b7e3ff2

I discovered another bug along the way in the pure-Python scanstring, int(s, 16) will parse '0xNN' when json expects only strings of the form 'NNNN' to work. I fixed that along with this issue by explicitly checking for x or X.

I forgot about issue11489. After reclassification this issue is it's duplicate.

Updated patch submitted in issue11489.