classification
Title: Refactor typing._GenericAlias
Type: enhancement Stage: resolved
Components: Library (Lib) Versions: Python 3.9
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Jelle Zijlstra, gvanrossum, levkivskyi, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2020-04-26 16:35 by serhiy.storchaka, last changed 2021-05-02 20:34 by Jelle Zijlstra. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 19719 merged serhiy.storchaka, 2020-04-26 16:37
PR 19984 merged serhiy.storchaka, 2020-05-07 18:45
PR 20021 merged serhiy.storchaka, 2020-05-10 09:39
Messages (12)
msg367312 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-04-26 16:35
typing._GenericAlias represents two different types: user defined (like List[int]) and special (like List). They have different behavior, and common methods contain special cases.

The proposed PR rewrites the implementation in more object-oriented paradigm: different classes for different behavior. _GenericAlias is split on three classes: user defined (it may be replaced with GenericAlias in future), special, and the base class for common methods. Its subclasses are also split on classes for special types Tuple and Callable and for parametrized Callable[] and Annotated[]. The work is not finished yet and the code is still complex.
msg368317 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-07 01:09
New changeset c1c7d8ead9eb214a6149a43e31a3213c52448877 by Serhiy Storchaka in branch 'master':
bpo-40397: Refactor typing._GenericAlias (GH-19719)
https://github.com/python/cpython/commit/c1c7d8ead9eb214a6149a43e31a3213c52448877
msg368367 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-07 18:44
The next PR removes __args__ and __parameters__ from _SpecialGenericAlias.

* No existing test is failed. It is an evidence that these attributes are not intended, but a side effect of mixing two different kinds in one class.

* get_args() ignores __args__ and returns () for _SpecialGenericAlias (after resolving issue40398).

* Nested _SpecialGenericAlias is not considered a generic class containing type variables:

>>> from typing import *
>>> T = TypeVar('T')
>>> Dict[int, List[T]][str]
typing.Dict[int, typing.List[str]]
>>> Dict[int, List][str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 695, in __getitem__
    _check_generic(self, params)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 194, in _check_generic
    raise TypeError(f"{cls} is not a generic class")
TypeError: typing.Dict[int, typing.List] is not a generic class

It also fixes the following error:

>>> from typing import *
>>> T = TypeVar('T')
>>> Dict[T, List][str]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 241, in inner
    return cached(*args, **kwds)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 703, in __getitem__
    subargs = tuple(subst[x] for x in arg.__parameters__)
  File "/home/serhiy/py/cpython-release/Lib/typing.py", line 703, in <genexpr>
    subargs = tuple(subst[x] for x in arg.__parameters__)
KeyError: ~T

and allows to simplify __eq__ and __hash__ for _SpecialGenericAlias. Currently it has a weird behavior in corner case:

>>> from typing import *
>>> from typing import T
>>> List == List[T]
True
msg368463 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2020-05-08 20:54
The most recent change here caused a regression. The following file:

```
from typing import Generic, TypeVar, Union

class CannotTransform(Exception): pass

T = TypeVar("T")
E = TypeVar("E", bound=Exception)

class Ok(Generic[T]): pass
class Err(Generic[E]): pass

Result = Union[Ok[T], Err[E]]
TResult = Result[T, CannotTransform]
TMatchResult = TResult[int]
```

Now fails with:

```
Traceback (most recent call last):
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/../../black/black.py", line 22, in <module>
    TMatchResult = TResult[int]
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 704, in __getitem__
    arg = arg[subargs]
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 244, in inner
    return func(*args, **kwds)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 695, in __getitem__
    _check_generic(self, params)
  File "/Users/jzijlstra-mpbt/py/cpython/Lib/typing.py", line 194, in _check_generic
    raise TypeError(f"{cls} is not a generic class")
TypeError: __main__.Err[__main__.CannotTransform] is not a generic class
```

Before commit c1c7d8ead9eb214a6149a43e31a3213c52448877 it was fine. This was found when we added 3.9-dev to CI for Black (https://github.com/psf/black/pull/1393).
msg368535 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-09 19:17
@Serhiy can you look at this? A possible fix might be:

diff --git a/Lib/typing.py b/Lib/typing.py
index 681ab6d21e..adcef1e82b 100644
--- a/Lib/typing.py
+++ b/Lib/typing.py
@@ -701,7 +701,8 @@ class _GenericAlias(_BaseGenericAlias, _root=True):
                 arg = subst[arg]
             elif isinstance(arg, (_BaseGenericAlias, GenericAlias)):
                 subargs = tuple(subst[x] for x in arg.__parameters__)
-                arg = arg[subargs]
+                if subargs:
+                    arg = arg[subargs]
             new_args.append(arg)
         return self.copy_with(tuple(new_args))
 

The immediate cause of the problem seems to be that we call arg[subargs] with subargs being (). So arg is a _GenericAlias but its __parameters__ is (). I'm not sure what path was taken previously. :-(
msg368556 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-10 00:48
I think I have a shorter repro, not involving unions:

from typing import *
T = TypeVar("T")
S = TypeVar("S")
U = TypeVar("U")
class A(Generic[T]): ...
class B(Generic[T]): ...
class C(Generic[T, S]): ...
print(C[A[U], B[int]][str])

Fails in the same place, works in 3.8 (I didn't check just before the offending commit), and the same fix works.
msg368570 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-10 08:53
New changeset fcb285609a2e55f2dc63dcfbb32e4e2fddf71546 by Serhiy Storchaka in branch 'master':
bpo-40397: Remove __args__ and __parameters__ from _SpecialGenericAlias (GH-19984)
https://github.com/python/cpython/commit/fcb285609a2e55f2dc63dcfbb32e4e2fddf71546
msg368571 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-10 09:04
Thank you Jelle for your report. There is even simpler example:

Dict[Tuple[T], List[int]][str]
msg368578 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-10 09:41
Even simpler: Dict[T, List[int]][str].

Dict[T, list[int]][str] also fails. But dict[T, list[int]][str] works as expected (and there are tests).
msg368583 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2020-05-10 10:39
New changeset 0122d48681b1df27015cf396559fb283ba364d6d by Serhiy Storchaka in branch 'master':
bpo-40397: Fix subscription of nested generic alias without parameters. (GH-20021)
https://github.com/python/cpython/commit/0122d48681b1df27015cf396559fb283ba364d6d
msg368586 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2020-05-10 14:32
Thanks Serhyi! I can confirm that the issue I posted is fixed.
msg392727 - (view) Author: Jelle Zijlstra (Jelle Zijlstra) * (Python triager) Date: 2021-05-02 20:34
Looks like there's nothing left to do here.
History
Date User Action Args
2021-05-02 20:34:09Jelle Zijlstrasetstatus: open -> closed
resolution: fixed
messages: + msg392727

stage: patch review -> resolved
2020-05-10 14:32:14Jelle Zijlstrasetpriority: release blocker -> normal

messages: + msg368586
2020-05-10 10:39:51serhiy.storchakasetmessages: + msg368583
2020-05-10 09:41:19serhiy.storchakasetmessages: + msg368578
2020-05-10 09:39:15serhiy.storchakasetpull_requests: + pull_request19332
2020-05-10 09:04:24serhiy.storchakasetmessages: + msg368571
2020-05-10 08:53:19serhiy.storchakasetmessages: + msg368570
2020-05-10 00:48:39gvanrossumsetmessages: + msg368556
2020-05-09 19:17:34gvanrossumsetmessages: + msg368535
2020-05-08 20:54:49Jelle Zijlstrasetpriority: normal -> release blocker
nosy: + Jelle Zijlstra
messages: + msg368463

2020-05-07 18:45:12serhiy.storchakasetpull_requests: + pull_request19300
2020-05-07 18:44:00serhiy.storchakasetmessages: + msg368367
2020-05-07 01:09:40serhiy.storchakasetmessages: + msg368317
2020-04-26 16:37:04serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request19040
2020-04-26 16:35:15serhiy.storchakacreate