Title: calling a _json.Encoder object raises a SystemError in case obj.items() returned a tuple
Type: behavior Stage: needs patch
Components: Extension Modules Versions: Python 3.6
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Oren Milman, berker.peksag, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2017-09-15 17:21 by Oren Milman, last changed 2017-11-07 22:35 by serhiy.storchaka.

Messages (5)
msg302278 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-09-15 17:21
the following code causes a SystemError:

import json.encoder
class BadDict(dict):
    def items(self):
        return ()

encoder = json.encoder.c_make_encoder(None, None, None, None, 'foo', 'bar',
                                      True, None, None)
encoder(obj=BadDict({'spam': 42}), _current_indent_level=4)

this is because encoder_call() (in Modules/_json.c) passes the 'obj' argument
so that eventually encoder_listencode_dict() calls PyMapping_Items() on it.
encoder_listencode_dict() assumes that PyMapping_Items() returned a list, and
passes it to PyList_Sort().

ISTM that subclassing dict and implementing items() so that it returns a tuple
is not unrealistic.

maybe we should silently convert the tuple that PyMapping_Items() returned to a
msg302290 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-09-15 18:53
Oh, I was aware of this issue for long time, but didn't have good opportunity for fixing it. And yes, making PyMapping_Items() (and friends) always returning a list looks like a reasonable option to me. This could fix similar bugs in third-party extensions. In Python 2 PyMapping_Items() is documented as returning a list, but actually it can return an arbitrary type. Authors of extensions could be fooled by the documentation and use concrete list API.

The drawback of this solution is some performance degradation in rare case of items() returning a tuple.
msg304712 - (view) Author: Oren Milman (Oren Milman) * Date: 2017-10-21 16:39
ISTM that PR 3840 resolved this issue (as part of bpo-28280).
msg305794 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2017-11-07 21:57
PR 3840 has been merged and it looks like Oren was correct. I'm getting the following output with current master:

>>> encoder(obj=BadDict({'spam': 42}), _current_indent_level=4)
msg305796 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-11-07 22:35
bpo-28280 fixed this issue only in the master branch. I had left this issue open for fixing 3.6.
Date User Action Args
2017-11-07 22:35:54serhiy.storchakasetstatus: closed -> open
resolution: out of date ->
messages: + msg305796

stage: resolved -> needs patch
2017-11-07 21:57:13berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg305794

resolution: out of date
stage: needs patch -> resolved
2017-10-21 17:20:38serhiy.storchakasetstage: needs patch
versions: + Python 3.6, - Python 3.7
2017-10-21 16:39:34Oren Milmansetmessages: + msg304712
2017-09-15 18:53:05serhiy.storchakasetnosy: + rhettinger, serhiy.storchaka
messages: + msg302290
2017-09-15 17:21:39Oren Milmancreate