classification
Title: [typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.10
Type: behavior Stage: patch review
Components: Interpreter Core Versions: Python 3.10
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: BTaskaya, barry, brett.cannon, eric.smith, gvanrossum, inada.naoki, levkivskyi, lukasz.langa
Priority: normal Keywords: patch

Created on 2019-10-27 16:07 by vstinner, last changed 2020-08-02 01:23 by inada.naoki.

Pull Requests
URL Status Linked Edit
PR 20434 open BTaskaya, 2020-05-26 16:06
Messages (21)
msg355476 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-27 16:07
The PEP 563: Postponed evaluation of annotations was introduced an opt-in feature using "from __future__ import annotations". It is scheduled to become the default in Python 4.0.

I would prefer to limit the number of incompatible changes in Python 4.0: it should just a "regular" release, with a regular number of incompatible changes. The version number change is going to cause enough troubles...

Would it be possible possible to enable postponed evaluation of annotations either before or after Python 4.0? For example, can we imagine to enable it by default in Python 3.9? If not, what about Python 3.10?

See also the PEP 608 (Coordinated Python release) and bpo-38604 (Schedule Py_UNICODE API removal).
msg355495 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2019-10-27 19:17
IMO 3.10 would be better, since 3.9 would be too soon (it would be like a schedule for a normal deprecation).

Also if we are really doing this, I think it is better to announce this soon.

Also we should try to fix relevant issues related to string annotations (in typing and dataclasses), like https://github.com/python/typing/issues/508, https://github.com/python/typing/issues/574, https://bugs.python.org/issue37838, https://bugs.python.org/issue34776 and https://bugs.python.org/issue37948.
msg355506 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-10-27 23:23
We never should have mentioned 4.0 as the target date to make this the
default (and only) behavior -- who knows whether there will ever even be a
Python 4.0? Even 3.10 might be on the early side (assuming we'll switch to
a year-long release cycle per PEP 602 -- we will then make deprecations in
general take two release cycles).

I do agree that we should start the process of deprecating the
non-__future__ behavior here in 3.9. I know of one project with a private
fork of Python (for other reasons) that has made this default.
msg355523 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-10-28 11:03
> We never should have mentioned 4.0 as the target date to make this the
default (and only) behavior

I am fine with modifying __future__ documentation to only modify the "Mandatory" column to remove Python 4.0, and then close this issue:
https://docs.python.org/dev/library/__future__.html

--

But I like the "PEP 563: Postponed evaluation of annotations", IMHO it would be nice to get it as the default behavior :-) It's just a matter of properly organize the transition ;-)
msg355546 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2019-10-28 14:43
You can bring the deprecation schedule up on discourse or python-dev so more folks can let us know whether they'd be okay with 3.9 or 3.10.
msg360830 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-01-28 02:19
I started a thread on typing-sig:
https://mail.python.org/archives/list/typing-sig@python.org/thread/JG4IWHO3TZORNESR6ARXFL6OUZU2BEBX/
msg366634 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2020-04-17 00:46
I personally like 3.10 as the target as that means users had at least 3 years to move to move over. Plus we can put a warning in the What's New for 3.9 about our plans for 3.10.
msg366635 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-17 00:50
This issue has been discussed during the Language Summit. A quick poll showed that the majority is in favor of changing the default in Python 3.9.

Lukasz proposed a PEP update to propose to switch the default in Python 3.9:
https://github.com/python/peps/pull/1371/

For me, the unclear part is which projects would be impacted if the default changes?

Someone mentioned attrs, but it seems like attrs is fine:
https://github.com/python-attrs/attrs/issues/288#issuecomment-587265961

In term of workflow, I would _prefer_ to get such incompatible in the very beginning of a devcycle, rather than just before the feature freeze. But I don't think that it's a blocker issue. Technically, changes are allowed until 3.9.0 beta1. Moreover, Lukasz is the 3.9 release manager, the author of the PEP 563 and he is in favor of changing the default in 3.9 :-)
msg369269 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-18 16:30
Too bad nobody took any action here after the positive outcome of the discussion at the summit.
msg369301 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-18 22:20
> Too bad nobody took any action here after the positive outcome of the discussion at the summit.

I didn't understand it this way. I understood that some people were not 100% comfortable to target 3.9. The question was 3.9 or 3.10. Since the release cycle is now shorter (1 years), only targeting 3.10 is not a big deal ;-)
msg369303 - (view) Author: Łukasz Langa (lukasz.langa) * (Python committer) Date: 2020-05-18 22:26
We'll make this an announced 3.10 feature early on. The discussion at the Summit wasn't as clear cut to me: 35% of participants would rather see this default later than 3.9.
msg370000 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 15:16
Is there anyone interested to implement this change in Python 3.10?
msg370020 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-05-26 17:24
I opened the PR 20434 as draft, but from what I understand, there is going to be some breakage (on our test suite). I'll try to narrow it down (currently ~4 tests instead of ~20) but I dont want to prevent anyone else from working on this, so feel free to ignore my PR if you have a working test suite with a low breakage level.
msg370028 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-05-26 20:27
After trying to complete a patch, there are a few issues that immediately showed itself (and this might lead to not to do this in 3.10, I dont know);

First one is double-forward-ref, which is usage of string-annotations when there is postponed evaluatation of annotations:
>>> import typing
>>> from __future__ import annotations
>>> def x(a: 'int'): pass
... 
>>> typing.get_type_hints(x)
{'a': ForwardRef('int')}

If we make annoatations feature default, this would be default behavior. The solution would be a workaround to the compiler;
static int
compiler_visit_annexpr(struct compiler *c, expr_ty annotation)
{
    if (annotation->kind == Constant_kind && PyUnicode_CheckExact(annotation->v.Constant.value)) {
        PyObject *text = annotation->v.Constant.value;
        Py_INCREF(text);
        ADDOP_LOAD_CONST_NEW(c, text);
    } else {
        ADDOP_LOAD_CONST_NEW(c, _PyAST_ExprAsUnicode(annotation));
    }
    return 1;
}
But I am not sure if this is too silly or not. 

The second problem is `inspect.signature`. If we don't resolve annotations there and continue it is definitely going to break some code. If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

The third problem is various dataclass hacks. Like `_type_{field.name}` etc. annotations and how ClassVar/InitVar parsed.

There are also some little parts that need to change. 

Any thoughts on these issues?
msg370031 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-05-26 20:45
> The second problem is `inspect.signature`. If we don't resolve annotations there and continue it is definitely going to break some code.

Would you mind to elaborate why would it break some code? Consumers of annotations should already be prepared to get directly types or strings, no?

> If we resolve, that would mean that annotations must able to point something real (and this might not be the real case if the user uses a string annotation etc.) and will break code. (both tried and both breaks different modules on the stdlib tests)

I expect that resolving has an impact on performance, whereas the caller may not use annotations at all but only cares of the number of parameters or their name.

It would be resonable to not resolve annotations in signature() by default. If someone cares, maybe a new parameter can be added to resolve annotations?
msg370035 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-26 22:18
- Double forward ref: IMO this can be resolved in the get_type_hints() functions. (Łukasz do you agree?)

- inspect.signature(): Maybe this could switch to using typing.get_type_hints()? Then again if performance is important here maybe we cannot change anything.

- dataclasses hacks: these should just be resolved.

Another thought: maybe some of these issues can be considered bugs in 3.9 as well, and we should fix them there too? That might help us decide the right path forward. After all we should really encourage people to start using `from __future__ import annotations` in their code, to help them get ready for these issues in 3.10.

Final thought: I know at least the Dropbox client team, and possibly also Instagram, has already turned on `from __future__ import annotations` by default in their local fork of Python. Maybe we can ask them if they ever felt the need to change inspect.signature or typing.get_type_hints.
msg370037 - (view) Author: Eric V. Smith (eric.smith) * (Python committer) Date: 2020-05-26 23:13
To my knowledge, dataclasses works with `from __future__ import annotations`. If there are specific examples of problems, I'd like to hear about it: please open a separate issue.

There is a hack (discussed at PyCon 2018 with all of the relevant players) where it avoids importing typing to look at typing.ClassVar, but I think that code is all correct. Maybe I should just bite the bullet and import typing, since I believe importing it is faster than it used to be.
msg370188 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-05-28 10:51
From now on, should typing.get_type_hints automatically resolve arguments too? An example would be this;

import typing
T = typing.TypeVar("T")
class Loop(typing.Generic[T]):
    subloop: typing.Final["Loop[int]"]
print(typing.get_type_hints(Loop))
>>> {'subloop': typing.Final[__main__.Loop[int]]}
If we run the same code under future annotations
>>> {'subloop': typing.Final[ForwardRef('Loop[int]')]}
msg370214 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-28 14:02
I think in general it is more insightful to discuss the behavior of get_type_hints() given specific things in annotations.

We generally don't write forward refs inside forward refs, like "SomeClass['int']". So maybe that code was wrong? Where did you find it?
msg370220 - (view) Author: Batuhan Taskaya (BTaskaya) * (Python triager) Date: 2020-05-28 14:31
An example would be this https://github.com/python/cpython/blob/24bddc1b3b58f6899b2d412e51b37f68536e4fe2/Lib/test/test_typing.py#L2744-L2745. Either I can change tests in order to reflect now everything is a forward ref by default
class Loop:
    attr: Final['Loop']
to
class Loop:
    attr: Final[Loop]
or resolve everything on get_type_hints.
msg370232 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2020-05-28 15:47
There will still be a lot of code written that way, because people need compatibility with earlier versions of Python. So I think it should be fixed in get_type_hints().
History
Date User Action Args
2020-08-02 01:23:39inada.naokisetnosy: + inada.naoki
2020-05-28 15:47:53gvanrossumsetmessages: + msg370232
2020-05-28 14:41:50vstinnersetnosy: - vstinner
2020-05-28 14:31:04BTaskayasetmessages: + msg370220
2020-05-28 14:02:42gvanrossumsetmessages: + msg370214
2020-05-28 10:51:15BTaskayasetmessages: + msg370188
2020-05-26 23:13:14eric.smithsetmessages: + msg370037
2020-05-26 22:18:57gvanrossumsetmessages: + msg370035
2020-05-26 20:45:55vstinnersetmessages: + msg370031
2020-05-26 20:27:06BTaskayasetmessages: + msg370028
2020-05-26 17:24:25BTaskayasetmessages: + msg370020
2020-05-26 17:21:47BTaskayasetmessages: - msg370003
2020-05-26 16:06:27BTaskayasetkeywords: + patch
stage: needs patch -> patch review
pull_requests: + pull_request19691
2020-05-26 15:42:15BTaskayasetmessages: + msg370003
2020-05-26 15:16:59vstinnersetversions: + Python 3.10
2020-05-26 15:16:53vstinnersetmessages: + msg370000
2020-05-18 22:26:31lukasz.langasetmessages: + msg369303
2020-05-18 22:20:54vstinnersetmessages: + msg369301
2020-05-18 16:30:06gvanrossumsetversions: - Python 3.9
messages: + msg369269

components: + Interpreter Core, - Library (Lib)
type: behavior
stage: needs patch
2020-05-18 15:40:30vstinnersettitle: [typing] PEP 563: Postponed evaluation of annotations: enable it by default before Python 4.0 -> [typing] PEP 563: Postponed evaluation of annotations: enable it by default in Python 3.10
2020-04-17 06:49:29BTaskayasetnosy: + BTaskaya
2020-04-17 00:50:37vstinnersetmessages: + msg366635
2020-04-17 00:46:45brett.cannonsetnosy: + brett.cannon
messages: + msg366634
2020-01-28 02:19:04vstinnersetmessages: + msg360830
2019-10-28 17:06:14barrysetnosy: + barry
2019-10-28 14:43:52gvanrossumsetmessages: + msg355546
2019-10-28 11:03:22vstinnersetmessages: + msg355523
2019-10-27 23:23:41gvanrossumsetmessages: + msg355506
2019-10-27 22:51:45eric.smithsetnosy: + eric.smith
2019-10-27 19:17:48levkivskyisetnosy: + gvanrossum
messages: + msg355495
2019-10-27 16:16:48xtreaksetnosy: + levkivskyi
2019-10-27 16:07:10vstinnercreate