Substitution of ParamSpec in Concatenate produces weird results:

>>> import typing
>>> P = typing.ParamSpec('P')
>>> typing.Concatenate[str, P][int]
typing.Concatenate[str, int]
>>> typing.Concatenate[str, P][[int]]
typing.Concatenate[str, (<class 'int'>,)]
>>> typing.Concatenate[str, P][typing.Concatenate[int, P]]
typing.Concatenate[str, typing.Concatenate[int, ~P]]

But all these results are invalid:

>>> typing.Concatenate[str, int]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, (int,)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, [int]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.
>>> typing.Concatenate[str, typing.Concatenate[int, P]]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/serhiy/py/cpython/Lib/typing.py", line 309, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 400, in __getitem__
    return self._getitem(self, parameters)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/serhiy/py/cpython/Lib/typing.py", line 595, in Concatenate
    raise TypeError("The last parameter to Concatenate should be a "
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: The last parameter to Concatenate should be a ParamSpec variable.

I expect that

1. The last parameter to Concatenate can be a Concatenate. Inner Concatenate should merge with the external one:

Concatenate[str, Concatenate[int, P]] -> Concatenate[str, int, P]
Concatenate[str, P][Concatenate[int, P]] -> Concatenate[str, int, P]

2. The last parameter to Concatenate can be a list of types. The result should be a list of types.

Concatenate[str, [int, dict]] -> [str, int, dict]
Concatenate[str, P][[int, dict]] -> [str, int, dict]

3. The last parameter to Concatenate can be an ellipsis.

Concatenate[str, ...]
Concatenate[str, P][...] -> Concatenate[str, ...]

Should Concatenate support substitution to begin with? PEP 612 doesn't say anything, and I am fairly certain it's a special typing form, not a generic. So I don't really understand what it means to substitute Concatenate.

Then again, Callable with a nested Concatenate can be substituted, and we currently implement that by using Concatenate's substitution behavior as proxy. But again, the meaning isn't clear to me.

I also noticed this strange part in PEP 612 about user-defined generic classes:

"`Generic[P]` makes a class generic on `parameters_expressions` (when P is a ParamSpec)":

...
class X(Generic[T, P]):
  f: Callable[P, int]
  x: T

def f(x: X[int, Concatenate[int, P_2]]) -> str: ...  # Accepted (KJ: What?)
...

The grammar for `parameters_expression` is:

parameters_expression ::=
  | "..."
  | "[" [ type_expression ("," type_expression)* ] "]"
  | parameter_specification_variable
  | concatenate "["
                   type_expression ("," type_expression)* ","
                   parameter_specification_variable
                "]"

I'm very confused. Does this mean Concatenate is valid when substituting user generics? Maybe I should ask the PEP authors?

My general sense when I implemented the PEP was that it was intended primarily for static type checking only. IMO, runtime correctness wasn't its concern.

My understanding is that type expression is valid when substitute TypeVar and parameters expression is valid when substitute ParamSpec.

New changeset ecfacc362dd7fef7715dcd94f2e2ca6c622ef115 by Serhiy Storchaka in branch 'main':
bpo-44791: Fix substitution of ParamSpec in Concatenate with different parameter expressions (GH-27518)
https://github.com/python/cpython/commit/ecfacc362dd7fef7715dcd94f2e2ca6c622ef115

New changeset 89db09029566cf3af04b540e33fe1ff9b32f8c8b by Miss Islington (bot) in branch '3.10':
bpo-44791: Fix substitution of ParamSpec in Concatenate with different parameter expressions (GH-27518)
https://github.com/python/cpython/commit/89db09029566cf3af04b540e33fe1ff9b32f8c8b

PR 27518 fixes a substitution of a ParamSpec variable with a Concatenate nad a list of types. It is not specified explicitly in PEP 612, but it does not contradict it.

PR 30969 makes an ellipsis be valid as the last argument of Concatenate to fix a substitution of a ParamSpec variable with an ellipsis. It contradicts with PEP 612 which allows only a ParamSpec variable there.

Before you start supporting things that contradict PEP 612 this should be discussed on typing-sig (or in the python/typing tracker).

Honestly I'd feel more comfortable if there was agreement on typing-sig with your previous PRs in this issue as well; "it does not contradict it" is not a ringing endorsement. :-)

I'm looking at https://github.com/python/cpython/pull/30969 and I'm not sure what the motivation for the change is. PEP 612 is quite precise here (https://www.python.org/dev/peps/pep-0612/#id1) and allows only a ParamSpec as the last argument to Concatenate.

What is the use case for using ... as the last argument? What should it mean to a static type checker?

Concatenate[int, ...] I would interpret as a function signature with first argument int, followed by arbitrary arguments afterwards. I haven't run into this case, but ... is allowed in other spots Paramspec is allowed currently.

P = Paramspec("P")

class Foo(Generic[P]):
  ...

Foo[...] # Allowed 
Callable[..., None] # Allowed

Are there any other places a paramspec is allowed? Generic type argument, first argument to callable, last to concatenate, anything else?

I'm unaware of any type checking use case for Concatenate[int, ...]. You can practically get same thing by using a paramspec variable without using it elsewhere so that it captures, but then effectively discards that information.

def use_func1(f: Callable[Concatenate[int, P], None]) -> None:
  ...

def use_func2(f: Callable[Concatenate[int, ...], None]) -> None:
  ...

feels like those two signatures should encode same information.