classification
Title: IntEnum is unpicklable by previous Python versions
Type: behavior Stage: resolved
Components: Versions: Python 3.4, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: ethan.furman Nosy List: barry, eli.bendersky, ethan.furman, python-dev, serhiy.storchaka
Priority: normal Keywords: 3.4regression, patch

Created on 2015-03-15 15:10 by ethan.furman, last changed 2015-03-19 01:29 by ethan.furman. This issue is now closed.

Files
File name Uploaded Description Edit
issue23673.stoneleaf.01.patch ethan.furman, 2015-03-16 17:59 review
Messages (4)
msg238148 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-03-15 15:10
IntEnum is advertised as being a drop-in replacement for integer contants; however this fails in the case of unpickling on previous Python versions.

This occurs because when a pickle is created the module, class, and value are stored -- but those don't exist prior to Python 3.4.

One solution is to modify IntEnum to pickle just like a plain int would, but this has the serious disadvantage of losing the benefits of IntEnum on Python versions that support it.

Another solution is to use Serhiy's idea of pickling by name; this would store the module and name to look up in that madule, and this works on all Python versions that have that constant: on Python 3.3 socket.AF_INET returns an integerer (2, I think), and on Python 3.4+ it returns the AF_INET AddressFamily member.
msg238235 - (view) Author: Ethan Furman (ethan.furman) * (Python committer) Date: 2015-03-16 17:59
Patch adds Enum._convert which is a class method that handles:

  - creating the new Enum
  - adding the appropriate members
  - adding the new Enum to the module's namespace (which is a passed parameter)
  - replacing the __reduce_ex__ method to return just the member's name

The change to Enum is:

    @classmethod
    def _convert(cls, name, module, filter, source=None):
        """
        Create a new Enum subclass that replaces a collection of global constants
        """
        # convert all constants from source (or module) that pass filter() to
        # a new Enum called name, and export the enum and its members back to
        # module;
        # also, replace the __reduce_ex__ method so unpickling works in
        # previous Python versions
        module_globals = vars(sys.modules[module])
        if source:
            source = vars(source)
        else:
            source = module_globals
        members = {name: value for name, value in source.items()
                if filter(name)}
        cls = cls(name, members, module=module)
        cls.__reduce_ex__ = _reduce_ex_by_name
        module_globals.update(cls.__members__)
        module_globals[name] = cls
        return cls

In use it looks like:

   IntEnum._convert(
        'AddressFamily',
        __name__,
        lambda C: C.isupper() and C.startswith('AF_'))

or

  _IntEnum._convert(
        '_SSLMethod', __name__,
        lambda name: name.startswith('PROTOCOL_'),
        source=_ssl)


ssl.py, socket.py, signal.py, and http/__init__.py have been updated to use this method.
msg238483 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-03-19 00:28
New changeset c93d46fd85ee by Ethan Furman in branch 'default':
issue23673
https://hg.python.org/cpython/rev/c93d46fd85ee
msg238486 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2015-03-19 01:24
New changeset 5dd7b5fb65e7 by Ethan Furman in branch '3.4':
issue23673
https://hg.python.org/cpython/rev/5dd7b5fb65e7
History
Date User Action Args
2015-03-19 01:29:05ethan.furmansetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2015-03-19 01:24:08python-devsetmessages: + msg238486
2015-03-19 00:35:04ethan.furmanlinkissue20680 superseder
2015-03-19 00:28:30python-devsetnosy: + python-dev
messages: + msg238483
2015-03-16 17:59:28ethan.furmansetfiles: + issue23673.stoneleaf.01.patch
keywords: + patch
messages: + msg238235

stage: test needed -> patch review
2015-03-15 15:10:45ethan.furmancreate