classification
Title: Add support for PEP 612 (Parameter Specification Variables) to typing.py
Type: behavior Stage: resolved
Components: Library (Lib) Versions: Python 3.10
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Guido.van.Rossum, gvanrossum, kj
Priority: normal Keywords: patch

Created on 2020-08-15 18:36 by gvanrossum, last changed 2021-04-28 16:14 by gvanrossum. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23702 merged kj, 2020-12-08 17:21
PR 24000 merged kj, 2020-12-29 17:34
PR 24056 closed kj, 2021-01-02 09:45
PR 25449 merged kj, 2021-04-17 02:46
Messages (22)
msg375487 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-08-15 18:36
We need stub versions of ParamSpec and Concatenate added to typing.py, plus tests that ensure these actually work in all situations required by the PEP. (It's not so important to ensure that they raise exceptions at runtime in cases where the PEP says they needn't work -- static type checkers will flag those better.)
msg382764 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-08 20:10
Thanks for working on this -- let me know when you have a question for me. Once this is ready we should also add it to the typing_extensions module so people can use it on older Python versions. (https://github.com/python/typing/tree/master/typing_extensions)
msg382778 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2020-12-08 23:44
I have a one question:
Should ParamSpec be treated as a type of TypeVar? This mostly pertains to
adding it to __parameters__ of generic aliases. Type substitution/chaining,
seems inappropiate for it right now.

Thanks for your help!
msg382805 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-09 21:33
I wrote

class C(Generic[T, P]): ...

and was surprised that C.__parameters__ was (T,) instead of (T, P).
msg382833 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2020-12-10 16:23
> and was surprised that C.__parameters__ was (T,) instead of (T, P).
Alright, I just fixed that :).

I'm now encountering many problems with how typing works which prevent what PEP 612 declare as valid from not throwing an error (these are all examples copied from the PEP)::

class X(Generic[T, P]): ...

1. X[int, [int, bool]] raises TypeError because second arg isn't a type (this is easy to fix).

2. X[int, ...] raises TypeError because for the same reason. One way to fix this is to automatically convert Ellipsis to EllipsisType just like how we already convert None to NoneType. Only problem now is that Callable[[...], int] doesn't raise TypeError. Should we just defer the problem of validating Callable to static type checkers instead?

class Z(Generic[P]): ...

3. Z[[int, str, bool]] raises TypeError, same as 1.

4. Z[int, str, bool] raises TypeError, but for too many parameters (not enough TypeVars). This one troubles me the most, current TypeVar substitution checks for len(__args__) == len(__parameters__).


I have a feeling your wish of greatly loosening type checks will come true ;). Should we proceed with that? That'd fix 1, 2, 3. The only showstopper now is 4. A quick idea is to just disable the check for len(__args__) == len(__parameters__) if there's only a single ParamSpec in __parameters__.
msg382837 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-10 17:58
> class X(Generic[T, P]): ...
> 
> 1. X[int, [int, bool]] raises TypeError because second arg isn't a type (this is easy to fix).

How would you fix it? By just allowing it? But then X.__args__ would be unhashable (see the other issue we're working on together).

> 2. X[int, ...] raises TypeError because for the same reason. One way to fix this is to automatically convert Ellipsis to EllipsisType just like how we already convert None to NoneType. Only problem now is that Callable[[...], int] doesn't raise TypeError. Should we just defer the problem of validating Callable to static type checkers instead?

I don't like using EllipsisType here, because this notation doesn't mean that we accept an ellipsis here -- it means (if I understand the PEP correctly) that we don't care about the paramspec.

> class Z(Generic[P]): ...
> 
> 3. Z[[int, str, bool]] raises TypeError, same as 1.

Same comment.

> 4. Z[int, str, bool] raises TypeError, but for too many parameters (not enough TypeVars). This one troubles me the most, current TypeVar substitution checks for len(__args__) == len(__parameters__).

The code should check that __parameters__ is (P,) for some ParamSpec P, and then transform this internally as if it was written Z[[int, str, bool]] and then typecheck that.

**BUT...**

How about an alternative implementation where as soon as we see that there's a ParamSpec in __parameters__ we just don't type-check at all, and accept anything? Leave it to the static type checker. Our goal here should be to support the syntax that PEP 612 describes so static type checkers can implement it, not to implement all the requirements described by the PEP at runtime.

I wonder what Pyre has in its stub files for ParamSpec -- maybe we can borrow that?
msg382843 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-10 20:30
For reference, here's what Pyre has (though it's an older version):

https://github.com/facebook/pyre-check/tree/master/pyre_extensions
msg382856 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2020-12-11 00:24
The pyre version in their __init__.py looks like they took your advice for
letting the static checker do the work wholeheartedly.

I'm not in favour of type checking either. Just that the pre-existing code
does it for me.

Not type checking when seeing ~P in __parameters__ would work, just that
__args__ will be unhashable (like you mentioned) so things will be slower
due to no type cache. Maybe we can cast the [int, str] to (int, str), that
should work with cache for most cases. And unlike the Callable issue -
since we don't need to ensure runtime correctness - we can ignore any weird
effects to __args__ and __parameters__ in ParamSpec, like TypeVars not
collecting etc.
msg382858 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-11 00:54
I started doing this in the original code (long ago when PEP 484 was brand
new) but have since realized that this makes the typing module both slow
and hard to maintain. We should not follow this example. I do think we
should try to keep `__args__` hashable, casting `[int, str]` to `(int,
str)` sounds right.
msg382938 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-13 19:42
This is now unblocked now that GH-23060 has landed.
msg383674 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-24 04:33
New changeset 73607be68668ab7f4bee53507c8dc7b5a46c9cb4 by kj in branch 'master':
bpo-41559: Implement PEP 612 - Add ParamSpec and Concatenate to typing (#23702)
https://github.com/python/cpython/commit/73607be68668ab7f4bee53507c8dc7b5a46c9cb4
msg383675 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-12-24 04:39
Huge thanks!

I think the next step is to port the essence to typing_extensions, which has to work for anything from 3.5 up (or maybe 3.6), *including* 3.10 and above.

https://github.com/python/typing/tree/master/typing_extensions

Honestly maybe we could just make ParamSpec an alias for TypeVar there, and make Concatenate an alias for Tuple?  Because "work" just means that the syntax needs to be valid, the type checkers are responsible for using it. (We're still working on getting this to work for mypy.)
msg383676 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2020-12-24 04:46
Thanks for the extremely helpful reviews and help in this Guido!

Sure, I'll probably start work on that next week, slightly busy with life right now. After that I'll work on docs.
msg384212 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-01-02 00:45
New changeset 11276cd9c49faea66ce7760f26a238d1edbf6421 by Ken Jin in branch 'master':
bpo-41559: Documentation for PEP 612 (GH-24000)
https://github.com/python/cpython/commit/11276cd9c49faea66ce7760f26a238d1edbf6421
msg384213 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-01-02 00:49
Looks like we can close this now, right? You can open a separate issue in the python/typing repo to update typing_extensions.
msg384217 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-01-02 04:30
I have just one more PR - one that converts genericalias nested lists to
tuples by default. It'll simplify the code a little bit and prepare 3.9.x
for PEP 612.
msg386903 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-02-13 04:41
Hi Guido, after a month of observing how people stumbled over the related collections.abc.Callable bugs, and experience from implementing this PEP, I learnt a few things which I think may interest you:

Previously it was *recommended* that everything in typing be hashable. I would now say it is *required*. Union uses sets to de-duplicate arguments, and Optional uses Union internally. Both blow up if things aren't hashable. A surprising number of people caught the collections.abc.Callable bug because Optional[Callable[.....]] failed.

Going forward, future PEPs to typing.py probably need to ensure their implementations are hashable, or risk not working with some of the types in the module itself. Alternatively, they can always change the implementations of Union and Optional, though I don't know if I recommend that ;).

Thanks for your time.
msg391269 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-04-17 03:10
Guido, I hope I didn't choose a bad time to send this PR over (I suspect you may already be flooded by emails about PEP 563).

The jist of the PR is that it's possible to implement PEP 612 in pure-Python. Specifically, PEP 612 says:

> As before, parameters_expressions by themselves are not acceptable in places where a type is expected

https://www.python.org/dev/peps/pep-0612/#valid-use-locations

Currently, the implementation treats ``ParamSpec`` specially in all builtin ``GenericAlias`` objects just for the sake of ``collections.abc.Callable``. There isn't a need for that - it just needs ``collections.abc.Callable`` to exhibit that behaviour.

By implementing this in pure Python, we can:
- Conform more strictly to the PEP.
- Reduce complexity of the builtin ``GenericAlias`` and also speed it up because less checks are required.
- Not tie more typing.py stuff to builtin ``GenericAlias``, which is a plus in my book.

What do you think?
msg391270 - (view) Author: Guido van Rossum (Guido.van.Rossum) Date: 2021-04-17 04:19
Yeah,  like this idea. Let’s get this in before beta 1.
msg392223 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-04-28 15:38
New changeset 859577c24981d6b36960d309f99f7fc810fe75c2 by Ken Jin in branch 'master':
bpo-41559: Change PEP 612 implementation to pure Python (#25449)
https://github.com/python/cpython/commit/859577c24981d6b36960d309f99f7fc810fe75c2
msg392228 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-04-28 15:47
The last of the patches have landed. Guido, thank you so much for helping me through this 5 month long process. Please enjoy your vacation!

PS: I need to send in a bugfix for typing.py later to ignore ``ParamSpec`` in the ``__parameters__`` of invalid locations like ``typing.List`` (this just ensures consistency with the builtin ``list``). That can be done as a bugfix patch after the 3.10 beta freeze (as it isn't a new feature at all). I'll open a new issue after beta 1 for that when you're back. Thanks!
msg392231 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2021-04-28 16:14
Thanks for your major contribution, Ken! Agreed, that bugfix can come later.
History
Date User Action Args
2021-04-28 16:14:24gvanrossumsetmessages: + msg392231
2021-04-28 15:47:23kjsetstatus: open -> closed
resolution: fixed
messages: + msg392228

stage: patch review -> resolved
2021-04-28 15:38:19gvanrossumsetmessages: + msg392223
2021-04-17 04:19:57Guido.van.Rossumsetnosy: + Guido.van.Rossum
messages: + msg391270
2021-04-17 03:10:35kjsetmessages: + msg391269
2021-04-17 02:46:16kjsetpull_requests: + pull_request24178
2021-02-13 04:41:43kjsetmessages: + msg386903
2021-01-02 09:45:37kjsetpull_requests: + pull_request22890
2021-01-02 04:30:10kjsetmessages: + msg384217
2021-01-02 00:49:59gvanrossumsetmessages: + msg384213
2021-01-02 00:45:58gvanrossumsetmessages: + msg384212
2020-12-29 17:34:32kjsetpull_requests: + pull_request22842
2020-12-24 04:46:19kjsetmessages: + msg383676
2020-12-24 04:39:01gvanrossumsetmessages: + msg383675
2020-12-24 04:33:56gvanrossumsetmessages: + msg383674
2020-12-13 19:42:06gvanrossumsetmessages: + msg382938
2020-12-11 21:15:25rhettingersettitle: Add support for PEP 612 to typing.py -> Add support for PEP 612 (Parameter Specification Variables) to typing.py
2020-12-11 21:14:55rhettingersettitle: Add support for PEP 612 to typing.py -> Add support for PEP 612 to typing.py
2020-12-11 00:54:35gvanrossumsetmessages: + msg382858
2020-12-11 00:24:12kjsetmessages: + msg382856
2020-12-10 20:30:05gvanrossumsetmessages: + msg382843
2020-12-10 17:58:29gvanrossumsetmessages: + msg382837
2020-12-10 16:23:58kjsetmessages: + msg382833
2020-12-09 21:33:10gvanrossumsetmessages: + msg382805
2020-12-08 23:44:35kjsetmessages: + msg382778
2020-12-08 20:10:43gvanrossumsetmessages: + msg382764
2020-12-08 17:21:11kjsetkeywords: + patch
nosy: + kj

pull_requests: + pull_request22569
stage: needs patch -> patch review
2020-08-15 18:36:04gvanrossumcreate