classification
Title: Pickling of typing types
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, levkivskyi, miss-islington, ned.deily, serhiy.storchaka, wrmsr
Priority: deferred blocker Keywords: patch

Created on 2018-02-19 10:46 by serhiy.storchaka, last changed 2018-04-05 00:46 by miss-islington. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 6216 merged levkivskyi, 2018-03-24 20:02
PR 6264 merged miss-islington, 2018-03-26 22:02
PR 6376 merged levkivskyi, 2018-04-04 22:30
PR 6378 merged miss-islington, 2018-04-05 00:25
Messages (15)
msg312346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-19 10:46
In 3.6 typing types are pickled by names:

>>> import pickle, pickletools, typing
>>> pickletools.optimize(pickle.dumps(typing.List))
b'\x80\x03ctyping\nList\n.'
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.List)))
    0: \x80 PROTO      3
    2: c    GLOBAL     'typing List'
   15: .    STOP
highest protocol among opcodes = 2

The side effect of this is that they are considered atomic by the copy module.

In 3.7 the pickle data contains all private attributes.

>>> pickletools.optimize(pickle.dumps(typing.List))
b'\x80\x03ctyping\n_GenericAlias\n)\x81}(X\x05\x00\x00\x00_inst\x89X\x08\x00\x00\x00_special\x88X\x05\x00\x00\x00_nameX\x04\x00\x00\x00ListX\n\x00\x00\x00__origin__cbuiltins\nlist\nX\x08\x00\x00\x00__args__ctyping\nTypeVar\n)\x81q\x00}(X\x04\x00\x00\x00nameX\x01\x00\x00\x00TX\x05\x00\x00\x00boundNX\x0b\x00\x00\x00constraints)X\x02\x00\x00\x00co\x89X\x06\x00\x00\x00contra\x89ub\x85X\x0e\x00\x00\x00__parameters__h\x00\x85X\t\x00\x00\x00__slots__Nub.'
>>> pickletools.dis(pickletools.optimize(pickle.dumps(typing.List)))
    0: \x80 PROTO      3
    2: c    GLOBAL     'typing _GenericAlias'
   24: )    EMPTY_TUPLE
   25: \x81 NEWOBJ
   26: }    EMPTY_DICT
   27: (    MARK
   28: X        BINUNICODE '_inst'
   38: \x89     NEWFALSE
   39: X        BINUNICODE '_special'
   52: \x88     NEWTRUE
   53: X        BINUNICODE '_name'
   63: X        BINUNICODE 'List'
   72: X        BINUNICODE '__origin__'
   87: c        GLOBAL     'builtins list'
  102: X        BINUNICODE '__args__'
  115: c        GLOBAL     'typing TypeVar'
  131: )        EMPTY_TUPLE
  132: \x81     NEWOBJ
  133: q        BINPUT     0
  135: }        EMPTY_DICT
  136: (        MARK
  137: X            BINUNICODE 'name'
  146: X            BINUNICODE 'T'
  152: X            BINUNICODE 'bound'
  162: N            NONE
  163: X            BINUNICODE 'constraints'
  179: )            EMPTY_TUPLE
  180: X            BINUNICODE 'co'
  187: \x89         NEWFALSE
  188: X            BINUNICODE 'contra'
  199: \x89         NEWFALSE
  200: u            SETITEMS   (MARK at 136)
  201: b        BUILD
  202: \x85     TUPLE1
  203: X        BINUNICODE '__parameters__'
  222: h        BINGET     0
  224: \x85     TUPLE1
  225: X        BINUNICODE '__slots__'
  239: N        NONE
  240: u        SETITEMS   (MARK at 27)
  241: b    BUILD
  242: .    STOP
highest protocol among opcodes = 2

Unpickling it creates a new object. And I'm not sure all invariants are satisfied.

In additional to lesses efficiency and lost of preserving identity, such pickle can be incompatible with old Python versions and future Python versions if the internal representation of typing types will be changed.
msg312354 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-02-19 16:21
I think it would be nice it would be pickled by name so the pickles are
compatible between Python versions.

What would we do for List[int]?

How are regular ABCs pickled?
msg312355 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-02-19 16:43
Here is the situation for 3.6 and before:

Generic classes are all actual class objects, so they are pickled as immutable. However this creates a problem, parameterized generics, such as `List[int]` _cannot_ be pickled in 3.6 and before, see https://github.com/python/typing/issues/511 (and this is something very hard to fix).

Here is the situation for 3.7:

Almost no generics are actual class objects, so they are pickled as usual. This also fixes the pickling problems in 3.6. However, there is one problematic thing, type variables, they should be pickled as immutable (i.e. by name reference), but I didn't have time to fix this, this is tracked in https://github.com/python/typing/issues/512

What is interesting this issue adds here is an idea that we can treat special typing aliases that are conceptually "unique" also as immutable. For example, `typing.List` will be pickled as "typing.List", while `typing.List[int]` will be pickled as _GenericAlias(<builtins.list>, args=(<builtins.int>,), ...)

Conveniently, all the special typing aliases are already marked with `_special=True`, so the potential solution would be like this:

class _GenericAlias:
    ...
    def __reduce__(self):
        if self._special:
            return 'typing.' + self._name
        return super().__reduce__()
msg312357 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-02-19 16:58
I think it would be better to pickle `typing.List[int]` as `operator.getitem(typing.List, int)`.

    def __reduce__(self):
        if self._special:
            return self._name # __module__ = 'typing'
        index = self._args
        if len(index) == 1:
            index, = index
        return operator.getitem, (self._unparametrized, index)

And there may be a special case for Union. I tried to implement this, but it seems to me that parametrized type doesn't have a reference to unparametrized type, and I don't know this code enough for writing idiomatic code.
msg312377 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-02-19 22:39
I'm honestly not too concerned about what happens with List[int] (though
doing a sensible thing here is not wrong :-), but I feel strongly that a
pickle containing a reference to typing.List should be compatible between
Python 3.6 and 3.7.
msg312952 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-02-26 20:47
So we need a decision on this about what, if anything, to do for 3.7.  The 3.7.0 ABI freeze is in 3.7.0b3; it would be better to get it resolved for 3.7.0b2.
msg312953 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-02-26 20:49
I am sick now, so can't work on this. There is a small chance I will be able to work on this issue this week. Is it possible to fix this in 3.7b3?
msg312954 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-02-26 20:52
> Is it possible to fix this in 3.7b3?

Yes.  Get well first!
msg312955 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-02-26 20:52
Thank you, Ned!
msg314482 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-03-26 22:01
New changeset 834940375ae88bc95794226dd8eff1f25fba1cf9 by Ivan Levkivskyi in branch 'master':
bpo-32873: Treat type variables and special typing forms as immutable by copy and pickle (GH-6216)
https://github.com/python/cpython/commit/834940375ae88bc95794226dd8eff1f25fba1cf9
msg314483 - (view) Author: miss-islington (miss-islington) Date: 2018-03-26 22:29
New changeset d0e04c82448c750d4dc27f2bddeddea74bd353ff by Miss Islington (bot) in branch '3.7':
bpo-32873: Treat type variables and special typing forms as immutable by copy and pickle (GH-6216)
https://github.com/python/cpython/commit/d0e04c82448c750d4dc27f2bddeddea74bd353ff
msg314952 - (view) Author: Will T (wrmsr) Date: 2018-04-04 21:47
I believe I hit a bug with this fix (just pulled the code a few min ago):

    In [10]: pickle.loads(pickle.dumps(typing.List))
    Out[10]: typing.List

    In [11]: pickle.loads(pickle.dumps(typing.FrozenSet))
    ---------------------------------------------------------------------------
    PicklingError                             Traceback (most recent call last)
    <ipython-input-11-be060c6090e3> in <module>()
    ----> 1 pickle.loads(pickle.dumps(typing.FrozenSet))

    PicklingError: Can't pickle typing.Frozenset: attribute lookup Frozenset on typing failed

The cause is in _GenericAlias.__init__

            name = orig_name[0].title() + orig_name[1:]

Maybe just pass the name explicitly?

For context I originally hit this trying to explicitly getattr(typing, alias_name) not by pickling but I'm pleased to see that's at least apparently intended to be valid use (I need to get the underlying special's parameter variance which is lost when you give it args).
msg314955 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-04-04 22:06
Apparently there is another type with a similar problem -- DefaultDict.

Will fix this now.
msg314963 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-04-05 00:25
New changeset 2a363d2930e29ec6d8a774973ed5a4965f881f5f by Ivan Levkivskyi in branch 'master':
bpo-32873: Remove a name hack for generic aliases in typing module (GH-6376)
https://github.com/python/cpython/commit/2a363d2930e29ec6d8a774973ed5a4965f881f5f
msg314964 - (view) Author: miss-islington (miss-islington) Date: 2018-04-05 00:46
New changeset 04eac02088f60192c7e54c7364bcaa892d7c05cf by Miss Islington (bot) in branch '3.7':
bpo-32873: Remove a name hack for generic aliases in typing module (GH-6376)
https://github.com/python/cpython/commit/04eac02088f60192c7e54c7364bcaa892d7c05cf
History
Date User Action Args
2018-04-05 00:46:42miss-islingtonsetmessages: + msg314964
2018-04-05 00:25:39miss-islingtonsetpull_requests: + pull_request6090
2018-04-05 00:25:17levkivskyisetmessages: + msg314963
2018-04-04 22:30:49levkivskyisetpull_requests: + pull_request6088
2018-04-04 22:06:30levkivskyisetmessages: + msg314955
2018-04-04 21:47:35wrmsrsetnosy: + wrmsr
messages: + msg314952
2018-03-26 22:37:30levkivskyisetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-03-26 22:29:14miss-islingtonsetnosy: + miss-islington
messages: + msg314483
2018-03-26 22:02:33miss-islingtonsetpull_requests: + pull_request5990
2018-03-26 22:01:20levkivskyisetmessages: + msg314482
2018-03-24 20:02:38levkivskyisetkeywords: + patch
stage: patch review
pull_requests: + pull_request5961
2018-02-26 20:52:47levkivskyisetmessages: + msg312955
2018-02-26 20:52:04ned.deilysetmessages: + msg312954
2018-02-26 20:49:50levkivskyisetmessages: + msg312953
2018-02-26 20:47:14ned.deilysetpriority: normal -> deferred blocker
nosy: + ned.deily
messages: + msg312952

2018-02-19 22:39:51gvanrossumsetmessages: + msg312377
2018-02-19 16:58:48serhiy.storchakasetmessages: + msg312357
2018-02-19 16:43:36levkivskyisetmessages: + msg312355
2018-02-19 16:21:21gvanrossumsetmessages: + msg312354
2018-02-19 10:46:45serhiy.storchakacreate