classification
Title: json.load failure when C optimizations aren't built
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.3, Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ezio.melotti Nosy List: alex, bob.ippolito, dstanek, ezio.melotti, fdrake, python-dev, terry.reedy
Priority: normal Keywords: patch

Created on 2010-07-12 15:47 by fdrake, last changed 2011-04-13 04:25 by ezio.melotti. This issue is now closed.

Files
File name Uploaded Description Edit
test-empty-json-object.diff fdrake, 2010-07-12 15:47 test that empty objects are converted correctly
issue9233.diff ezio.melotti, 2011-04-12 12:39 Patch to skip the tests when _json is not available (2.7) review
Messages (10)
msg110099 - (view) Author: Fred L. Drake, Jr. (fdrake) (Python committer) Date: 2010-07-12 15:47
The json decoder doesn't pass tests when the C optimizations in the _json module aren't available.  The actual test failures appear fairly superficial, but aren't the limit of the problems.

Empty objects are converted to empty lists instead of empty dictionaries when _json isn't available; test attached.

Issue 5723 notes that the pure-Python implementation isn't tested; fixing that might have avoided these problems.

Test failures from Python 2.6.5:

======================================================================
ERROR: test_c_encode_basestring_ascii (json.tests.test_encode_basestring_ascii.TestEncodeBaseStringAscii)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_encode_basestring_ascii.py", line 29, in test_c_encode_basestring_ascii
    self._test_encode_basestring_ascii(json.encoder.c_encode_basestring_ascii)
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_encode_basestring_ascii.py", line 32, in _test_encode_basestring_ascii
    fname = encode_basestring_ascii.__name__
AttributeError: 'NoneType' object has no attribute '__name__'

======================================================================
ERROR: test_c_scanstring (json.tests.test_scanstring.TestScanString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_scanstring.py", line 13, in test_c_scanstring
    self._test_scanstring(json.decoder.c_scanstring)
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_scanstring.py", line 17, in _test_scanstring
    scanstring('"z\\ud834\\udd20x"', 1, None, True),
TypeError: 'NoneType' object is not callable

======================================================================
FAIL: test_encode_basestring_ascii (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_speedups.py", line 13, in test_encode_basestring_ascii
    self.assertEquals(encoder.encode_basestring_ascii.__module__, "_json")
AssertionError: 'json.encoder' != '_json'

======================================================================
FAIL: test_scanstring (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.6.5/Lib/json/tests/test_speedups.py", line 9, in test_scanstring
    self.assertEquals(decoder.scanstring.__module__, "_json")
AssertionError: 'json.decoder' != '_json'


Test failures from 2.7:

======================================================================
ERROR: test_c_scanstring (json.tests.test_scanstring.TestScanString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.7/Lib/json/tests/test_scanstring.py", line 13, in test_c_scanstring
    self._test_scanstring(json.decoder.c_scanstring)
  File "/home/fdrake/src/Python-2.7/Lib/json/tests/test_scanstring.py", line 17, in _test_scanstring
    scanstring('"z\\ud834\\udd20x"', 1, None, True),
TypeError: 'NoneType' object is not callable

======================================================================
FAIL: test_encode_basestring_ascii (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.7/Lib/json/tests/test_speedups.py", line 12, in test_encode_basestring_ascii
    self.assertEquals(encoder.encode_basestring_ascii.__module__, "_json")
AssertionError: 'json.encoder' != '_json'

======================================================================
FAIL: test_scanstring (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/fdrake/src/Python-2.7/Lib/json/tests/test_speedups.py", line 8, in test_scanstring
    self.assertEquals(decoder.scanstring.__module__, "_json")
AssertionError: 'json.decoder' != '_json'
msg110111 - (view) Author: Fred L. Drake, Jr. (fdrake) (Python committer) Date: 2010-07-12 16:51
The attached test only fails for Python 2.7; removing Python 2.6 from this issue.
msg127166 - (view) Author: Bob Ippolito (bob.ippolito) * (Python committer) Date: 2011-01-27 00:48
For what it's worth, I don't think any of the failures mentioned here are in the current version of simplejson. There aren't any tests that check the speedup module name, the speedup tests are skipped when speedups are not detected, and the empty json object bug was fixed in June 2009.

http://code.google.com/p/simplejson/issues/detail?id=57
msg127355 - (view) Author: Alex Gaynor (alex) * (Python committer) Date: 2011-01-28 22:03
Just a note from downstream with PyPy: we've cherry-picked the commit Bob linked into our copy of the 2.7 stdlib since we don't have an _json (yet).
msg133574 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-04-12 12:39
I ran the tests again on 2.7 and got 3 failures:

======================================================================
ERROR: test_c_scanstring (json.tests.test_scanstring.TestScanString)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wolf/dev/py/issue9233/Lib/json/tests/test_scanstring.py", line 13, in test_c_scanstring
    self._test_scanstring(json.decoder.c_scanstring)
  File "/home/wolf/dev/py/issue9233/Lib/json/tests/test_scanstring.py", line 17, in _test_scanstring
    scanstring('"z\\ud834\\udd20x"', 1, None, True),
TypeError: 'NoneType' object is not callable

======================================================================
FAIL: test_encode_basestring_ascii (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wolf/dev/py/issue9233/Lib/json/tests/test_speedups.py", line 14, in test_encode_basestring_ascii
    self.assertEqual(encoder.encode_basestring_ascii.__module__, "_json")
AssertionError: 'json.encoder' != '_json'

======================================================================
FAIL: test_scanstring (json.tests.test_speedups.TestSpeedups)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wolf/dev/py/issue9233/Lib/json/tests/test_speedups.py", line 11, in test_scanstring
    self.assertTrue(decoder.scanstring is decoder.c_scanstring)
AssertionError: False is not true


The attached patch checks if _json is importable and skip the tests if it's not.
msg133579 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-12 13:07
New changeset 500063f6ae5a by Ezio Melotti in branch '2.7':
#9233: skip _json-specific tests when _json is not available.
http://hg.python.org/cpython/rev/500063f6ae5a
msg133597 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-04-12 18:59
Do those new check finish this issue and should it be closed?
If not, what is left?
msg133639 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-13 02:38
New changeset d58c63ff5bb2 by Ezio Melotti in branch '2.7':
#9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
http://hg.python.org/cpython/rev/d58c63ff5bb2
msg133641 - (view) Author: Roundup Robot (python-dev) Date: 2011-04-13 04:21
New changeset 36e676a8541d by Ezio Melotti in branch '3.1':
#9233: skip _json-specific tests when _json is not available.
http://hg.python.org/cpython/rev/36e676a8541d

New changeset 7019fc1a9663 by Ezio Melotti in branch '3.1':
#9233: Fix json to work properly even when _json is not available.
http://hg.python.org/cpython/rev/7019fc1a9663

New changeset a220458179ed by Ezio Melotti in branch '3.1':
#9233: Fix json.loads({}) to return a dict (instead of a list), when _json is not available.
http://hg.python.org/cpython/rev/a220458179ed
msg133642 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2011-04-13 04:25
Fixed in 2.7 and 3.1 and ported to 3.2 (b279611146d7) and 3.3 (e8e3f2b72a32).
In 3.1 json was completely broken without _json, so I fixed that too.
I also did some minor cleanup in ec6d881f5b02.
History
Date User Action Args
2011-04-13 04:25:47ezio.melottisetstatus: open -> closed
versions: + Python 3.1, Python 3.2, Python 3.3
messages: + msg133642

resolution: fixed
stage: needs patch -> resolved
2011-04-13 04:21:52python-devsetmessages: + msg133641
2011-04-13 02:38:17python-devsetmessages: + msg133639
2011-04-12 18:59:14terry.reedysetnosy: + terry.reedy
messages: + msg133597
2011-04-12 13:09:01ezio.melottisetassignee: ezio.melotti
versions: - Python 2.6
2011-04-12 13:07:28python-devsetnosy: + python-dev
messages: + msg133579
2011-04-12 12:39:29ezio.melottisetfiles: + issue9233.diff

messages: + msg133574
2011-01-28 22:03:22alexsetnosy: + alex
messages: + msg127355
2011-01-27 00:48:50bob.ippolitosetnosy: fdrake, bob.ippolito, dstanek, ezio.melotti
messages: + msg127166
2011-01-26 17:54:34dstaneksetnosy: + dstanek
2010-07-12 16:51:36fdrakesetmessages: + msg110111
2010-07-12 15:49:18ezio.melottisetnosy: + bob.ippolito, ezio.melotti

stage: needs patch
2010-07-12 15:47:44fdrakecreate