We have a pytest that boils down to the following:

#from __future__ import annotations
from inspect import Parameter, signature


def foo(x: str) -> str:
    return x + x


def test_foo():
    expected = (
        Parameter("x", Parameter.POSITIONAL_OR_KEYWORD, annotation=str),
    )

    actual = tuple(signature(foo).parameters.values())

    assert expected == actual

(execute with pip install pytest && pytest -vv test_foo.py)

I tried importing 3.10 annotations (so that we can get rid of quotes around the class containing foo(), which is omitted here because it isn't necessary to reproduce the bug), but doing so changes the output of inspect.signature() but not the output inspect.Parameter(), causing a mismatch between the two that breaks the test.

The above passes on 3.7.9, 3.8.7 & 3.9.1, and if I uncomment the first line, it fails on those same versions.
As can be expected, the annotations import is a no-op on 3.10.0a5 and the test passes either way.

I expect inspect might have not been correctly updated to support postponed annotations, but I haven't looked at the implementation (I'm not familiar with the CPython codebase at all) so it's just a guess.

Can you show the values of “expected” and “actual” for both the failures and successes?

Sure thing.

Putting the reproducing code in test_foo.py and running docker run --rm -it -v $PWD:/code python sh -c 'pip install pytest && pytest -vvv /code/test_foo.py' yields:

Collecting pytest
  Downloading pytest-6.2.2-py3-none-any.whl (280 kB)
     |████████████████████████████████| 280 kB 516 kB/s
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting py>=1.8.2
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.6 MB/s
Collecting attrs>=19.2.0
  Downloading attrs-20.3.0-py2.py3-none-any.whl (49 kB)
     |████████████████████████████████| 49 kB 1.8 MB/s
Collecting iniconfig
  Downloading iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 833 kB/s
Collecting pluggy<1.0.0a1,>=0.12
  Downloading pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 1.5 MB/s
Installing collected packages: pyparsing, toml, py, pluggy, packaging, iniconfig, attrs, pytest
Successfully installed attrs-20.3.0 iniconfig-1.1.1 packaging-20.9 pluggy-0.13.1 py-1.10.0 pyparsing-2.4.7 pytest-6.2.2 toml-0.10.2
============================= test session starts ==============================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /usr/local/bin/python
cachedir: .pytest_cache
rootdir: /code
collected 1 item

code/test_foo.py::test_foo FAILED                                        [100%]

=================================== FAILURES ===================================
___________________________________ test_foo ___________________________________

    def test_foo():
        expected = (
            Parameter("x", Parameter.POSITIONAL_OR_KEYWORD, annotation=str),
        )

        actual = tuple(signature(foo).parameters.values())

>       assert expected == actual
E       assert (<Parameter "x: str">,) == (<Parameter "x: 'str'">,)
E         At index 0 diff: <Parameter "x: str"> != <Parameter "x: 'str'">
E         Full diff:
E         - (<Parameter "x: 'str'">,)
E         ?                 -   -
E         + (<Parameter "x: str">,)

code/test_foo.py:16: AssertionError
=========================== short test summary info ============================
FAILED code/test_foo.py::test_foo - assert (<Parameter "x: str">,) == (<Param...
============================== 1 failed in 0.02s ===============================

@eric.smith, here's a summarized version. I hope it helps:

def foo(x: str) -> str: ...

In Python 3.7 - 3.9 with from __future__ import annotations, inspect.signature sees foo's parameter as:

<Parameter "x: 'str'">

Without the future import (and also in Python 3.10):

<Parameter "x: str">

The reason for the difference in 3.10 (IIRC) is that inspect.signature auto converts all strings to typing.ForwardRef internally and then resolves them. This is a result of PEP-563 becoming default (also mentioned here https://docs.python.org/3.10/whatsnew/3.10.html#pep-563-postponed-evaluation-of-annotations-becomes-default )

Prior to 3.10, the future import treats function annotations as strings and inspect.signature _doesn't_ convert them. I don't know of this is a bug or intentional. Especially considering what PEP-563 has to say: https://www.python.org/dev/peps/pep-0563/#resolving-type-hints-at-runtime

@1ace, a possible workaround if you want full compatibility regardless of the future import is to use typing.get_type_hints:

(The result doesn't change even with from __future__ import annotations. This also works from 3.7-3.10)
>>> from typing import get_type_hints
>>> get_type_hints(foo)
{'x': <class 'str'>, 'return': <class 'str'>}

Sorry, I just realized you asked for the "success" value as well.

For python 3.7-3.9 without the import annotations, and for python 3.10, the type annotation is a class, ie. <Parameter "x: str">.

(... and I had the window open for too long and didn't notice Ken's reply before mine ^^')

@kj: thanks for the workaround! it's good to be able to change our code and not have to depend on a yet-to-be-released patch on each python version :)

That said, typing.get_type_hints() only returns the type hint, whereas our current test checks for other things (like the default value) which need to match (I think) for cython's use of __signature__? (I inherited all this code, so I'm not familiar with it and its history, and I'm beginning to question this test's necessity.)

I think this is all about: should inspect.signature() resolve string annotations into actual types (via get_type_hints, or whatever)? I don't use inspect much, so I can't offer an opinion there.

3.7 only gets security fixes

Python 3.10 will add an eval_str= argument to inspect.signature() that lets you evaluate string annotations. I don't think it makes sense to change anything in the bugfix branches, so I propose that this issue be closed.