Title: 2.7 json encoding broken for enums
Type: behavior Stage: resolved
Components: Documentation Versions: Python 2.7
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: docs@python Nosy List: barry, berker.peksag, docs@python, eddygeek, eli.bendersky, ezio.melotti, pitrou, rhettinger, serhiy.storchaka
Priority: normal Keywords:

Created on 2014-08-29 11:00 by eddygeek, last changed 2016-04-19 13:41 by berker.peksag. This issue is now closed.

Messages (8)
msg226057 - (view) Author: Edward O (eddygeek) * Date: 2014-08-29 11:00
_make_iterencode in python2.7/json/ encodes custom enum types incorrectly (the label will be printed without '"') because of these lines (line 320 in 2.7.6):
    elif isinstance(value, (int, long)):
        yield buf + str(value)

in constract, _make_iterencode in python 3 explicitly supports the enum types:

    elif isinstance(value, int):
        # Subclasses of int/float may override __str__, but we still
        # want to encode them as integers/floats in JSON. One example
        # within the standard library is IntEnum.
        yield buf + str(int(value))
msg226141 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-08-30 17:39
The enum types was added to the stdlib in 3.4. There are no the enum types in Python 2.7. There is no a bug, support for the enum types is new feature.
msg226142 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-08-30 17:45
Edward, is this a regression? If yes, we should probably fix it.
msg226145 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-08-30 19:34
This is not a regression.  json only deals with standard types (int, str, etc.), and Enum is not a standard type.

Enum was introduced in 3.4, so corresponding changes were made to json to support int and float subclasses, of which IntEnum is one.

In other words, this was a bug that no one noticed for many many releases, and I'm not sure we should fix it in 2.7 now.

Arguments for fixing?
msg226147 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2014-08-30 19:51
One argument against fixing:  If we do fix in 2.7.9 then any program targeting it will be unable to target 3.0-3.3, as those versions do not have the fix from 3.4.
msg226148 - (view) Author: Barry A. Warsaw (barry) * (Python committer) Date: 2014-08-30 20:26
On Aug 30, 2014, at 07:34 PM, Ethan Furman wrote:

>In other words, this was a bug that no one noticed for many many releases,
>and I'm not sure we should fix it in 2.7 now.
>Arguments for fixing?

-1 on fixing it, but we *can* document workarounds.  Here's what I use in
Mailman 3.

class ExtendedEncoder(json.JSONEncoder):
    """An extended JSON encoder which knows about other data types."""

    def default(self, obj):
        if isinstance(obj, datetime):
            return obj.isoformat()
        elif isinstance(obj, timedelta):
            # as_timedelta() does not recognize microseconds, so convert these
            # to floating seconds, but only if there are any seconds.
            if obj.seconds > 0 or obj.microseconds > 0:
                seconds = obj.seconds + obj.microseconds / 1000000.0
                return '{0}d{1}s'.format(obj.days, seconds)
            return '{0}d'.format(obj.days)
        elif isinstance(obj, Enum):
            # It's up to the decoding validator to associate this name with
            # the right Enum class.
        return json.JSONEncoder.default(self, obj)

(Frankly, I wish it was easier to extend the encoder, e.g. by registering
callbacks for non-standard types.)

I don't automatically decode enums because on PUTs, POSTs, and PATCHs, I know
which attributes should be enums, so I can convert them explicitly when I
validate input forms.
msg228770 - (view) Author: Edward O (eddygeek) * Date: 2014-10-07 15:50
The arguments for fixing:

* int subclasses overriding str is a very common usecase for enums (so much so that it was added to stdlib in 3.4).

* json supporting a standard type of a subsequent python version, though not mandatory, would be beneficial to Py2/Py3 compatibility.

* fixing this cannot break existing code

* the fix could theoretically be done to 3.0-3.3 if Ethan's argument is deemed important.
msg263736 - (view) Author: Berker Peksag (berker.peksag) * (Python committer) Date: 2016-04-19 13:41
-1 from me, too.
Date User Action Args
2016-04-19 13:41:05berker.peksagsetstatus: open -> closed

nosy: + berker.peksag
messages: + msg263736

stage: resolved
2015-07-21 07:57:34ethan.furmansetnosy: - ethan.furman
2014-10-07 15:50:44eddygeeksetstatus: pending -> open

messages: + msg228770
2014-10-05 10:55:29serhiy.storchakasetstatus: open -> pending

nosy: + docs@python
assignee: docs@python
components: + Documentation, - Library (Lib)
stage: test needed -> (no value)
2014-08-30 20:26:22barrysetmessages: + msg226148
2014-08-30 19:51:09ethan.furmansetmessages: + msg226147
2014-08-30 19:34:55ethan.furmansetmessages: + msg226145
2014-08-30 17:45:29pitrousetstatus: pending -> open

messages: + msg226142
2014-08-30 17:39:11serhiy.storchakasetstatus: open -> pending

nosy: + ethan.furman, barry, eli.bendersky, serhiy.storchaka
messages: + msg226141

resolution: not a bug
2014-08-29 20:05:02terry.reedysetnosy: + rhettinger, pitrou, ezio.melotti

stage: test needed
2014-08-29 11:00:51eddygeeksettitle: json encoding broken for -> 2.7 json encoding broken for enums
2014-08-29 11:00:34eddygeekcreate