classification
Title: Substitution of ParamSpec in Concatenate
Type: Stage: patch review
Components: Library (Lib) Versions: Python 3.11, Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: gvanrossum, kj, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2021-07-31 10:25 by serhiy.storchaka, last changed 2021-07-31 20:21 by serhiy.storchaka.

Pull Requests
URL Status Linked Edit
PR 27518 open serhiy.storchaka, 2021-07-31 20:21
Messages (3)
msg398632 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-07-31 10:25
Substitution of ParamSpec in Concatenate produces weird results:

>>> import typing
>>> P = typing.ParamSpec('P')
>>> typing.Concatenate[str, P][int]
typing.Concatenate[str, int]
>>> typing.Concatenate[str, P][[int]]
typing.Concatenate[str, (<class 'int'>,)]
>>> typing.Concatenate[str, P][typing.Concatenate[int, P]]
typing.Concatenate[str, typing.Concatenate[int, ~P]]

But all these results are invalid:

>>> typing.Concatenate[str, int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, (int,)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, [int]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, typing.Concatenate[int, P]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.

I expect that

1. The last parameter to Concatenate can be a Concatenate. Inner Concatenate should merge with the external one:

Concatenate[str, Concatenate[int, P]] -> Concatenate[str, int, P]
Concatenate[str, P][Concatenate[int, P]] -> Concatenate[str, int, P]

2. The last parameter to Concatenate can be a list of types. The result should be a list of types.

Concatenate[str, [int, dict]] -> [str, int, dict]
Concatenate[str, P][[int, dict]] -> [str, int, dict]

3. The last parameter to Concatenate can be an ellipsis.

Concatenate[str, ...]
Concatenate[str, P][...] -> Concatenate[str, ...]
msg398645 - (view) Author: Ken Jin (kj) * (Python committer) Date: 2021-07-31 16:21
Should Concatenate support substitution to begin with? PEP 612 doesn't say anything, and I am fairly certain it's a special typing form, not a generic. So I don't really understand what it means to substitute Concatenate.

Then again, Callable with a nested Concatenate can be substituted, and we currently implement that by using Concatenate's substitution behavior as proxy. But again, the meaning isn't clear to me.

I also noticed this strange part in PEP 612 about user-defined generic classes:

"`Generic[P]` makes a class generic on `parameters_expressions` (when P is a ParamSpec)":

...
class X(Generic[T, P]):
  f: Callable[P, int]
  x: T

def f(x: X[int, Concatenate[int, P_2]]) -> str: ...  # Accepted (KJ: What?)
...

The grammar for `parameters_expression` is:

parameters_expression ::=
  | "..."
  | "[" [ type_expression ("," type_expression)* ] "]"
  | parameter_specification_variable
  | concatenate "["
                   type_expression ("," type_expression)* ","
                   parameter_specification_variable
                "]"

I'm very confused. Does this mean Concatenate is valid when substituting user generics? Maybe I should ask the PEP authors?

My general sense when I implemented the PEP was that it was intended primarily for static type checking only. IMO, runtime correctness wasn't its concern.
msg398661 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2021-07-31 19:27
My understanding is that type expression is valid when substitute TypeVar and parameters expression is valid when substitute ParamSpec.
History
Date User Action Args
2021-07-31 20:21:09serhiy.storchakasetkeywords: + patch
stage: patch review
pull_requests: + pull_request26033
2021-07-31 19:27:18serhiy.storchakasetmessages: + msg398661
2021-07-31 16:21:23kjsetmessages: + msg398645
2021-07-31 10:25:14serhiy.storchakacreate