This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Can't create generic NamedTuple as of py3.9
Type: behavior Stage: patch review
Components: Versions: Python 3.11, Python 3.10, Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: AlexWaygood, FHTMitchell, JelleZijlstra, Steven Silvester, behackett, dlukes, eric.smith, graingert, gvanrossum, juliusgeo, kj, levkivskyi, python-dev, rhettinger, serhiy.storchaka, sobolevn
Priority: normal Keywords: patch

Created on 2021-04-23 16:22 by FHTMitchell, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 31679 closed python-dev, 2022-03-04 09:40
PR 31779 open juliusgeo, 2022-03-09 04:16
PR 31781 open serhiy.storchaka, 2022-03-09 10:16
Messages (25)
msg391705 - (view) Author: FHTMitchell (FHTMitchell) Date: 2021-04-23 16:22
As of python 3.9, you now can't have multiple inheritance with `typing.NamedTuple` subclasses. This seems sensible, until you realise that `typing.Generic` works via inheritance. This fails whether or not `from __future__ import annotations` is enabled.

example:

```
class Group(NamedTuple, Generic[T]):
     key: T
     group: List[T]
 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-fc238c4826d7> in <module>
----> 1 class Group(NamedTuple, Generic[T]):
      2     key: T
      3     group: List[T]
      4 

~/.conda/envs/py39/lib/python3.9/typing.py in _namedtuple_mro_entries(bases)
   1818 def _namedtuple_mro_entries(bases):
   1819     if len(bases) > 1:
-> 1820         raise TypeError("Multiple inheritance with NamedTuple is not supported")
   1821     assert bases[0] is NamedTuple
   1822     return (_NamedTuple,)

TypeError: Multiple inheritance with NamedTuple is not supported
```

This worked fine in python 3.7 and 3.8 and as I understand it was one of the motivating cases for pep 560. 

The change was made as part of bpo-40185: Refactor typing.NamedTuple. Whilst the obvious alternative is "use dataclasses", they don't have the same runtime properties or implications as namedtuples.
msg406083 - (view) Author: David Lukeš (dlukes) * Date: 2021-11-10 11:48
This is unfortunate, especially since it used to work... Going forward, is the intention not to support this use case? Or is it possible that support for generic NamedTuples will be re-added in the future?
msg406124 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2021-11-10 18:57
+1 for reverting this change and restoring the previous behavior.
msg414541 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-04 17:54
Couldn't there be a subtler solution than rolling back GH-19371?
msg414557 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-05 01:40
Was this ever documented to work? We have now disallowed this behavior in 3.9 and 3.10 with few complaints, so it doesn't seem that important to restore it. Also, static type checkers generally disallow generic NamedTuples.
msg414558 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-05 02:55
Mypy seems to allow this:

from typing import NamedTuple, TypeVar, Generic, List, Tuple

T = TypeVar("T")

class New(NamedTuple, Generic[T]):
    x: List[T]
    y: Tuple[T, T]

It's true that pyright doesn't, but maybe that's because it doesn't work in 3.9-3.10?
msg414559 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-05 02:57
It doesn't really. If you do `x = New([1], (2, 3))` you get:

main.py:11: error: List item 0 has incompatible type "int"; expected "T"
main.py:11: error: Argument 2 to "New" has incompatible type "Tuple[int, int]"; expected "Tuple[T, T]"

https://mypy-play.net/?mypy=latest&python=3.10&gist=a13c7a33c55a3aeee95324d46cd03ffd
msg414563 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-05 05:12
So if it doesn't work in mypy why bother making it work at runtime? Is there an actual use case that broke? (There might be, but probably esoteric if nobody's run into it until now.)
msg414567 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-05 08:26
I actually have quite a few use cases for this feature. It's true that type checkers don't (yet) support it, but that doesn't mean that it should be disallowed at runtime. In fact, allowing it at runtime will surely give type  checkers room to experiment with implementing this feature if it is requested by enough users. As it is, they are blocked from doing so.
msg414568 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-05 08:28
> Is there an actual use case that broke?

No, because this was never usable in the first place. But there are those who wish it were usable :)
msg414570 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-05 09:06
Please don't revert all changes. It will not make it working right in any case. See issue44863 for more correct approach.
msg414584 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-05 16:08
Can you be more specific about your use cases?
msg414585 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-05 16:49
Consider the typeshed stub for `concurrent.futures.DoneAndNotDoneFutures`. At runtime this is a `collections.namedtuple`, but in the stub, we need it to be generic to allow precise type inference. But we can't have a generic NamedTuple, so the stub is currently this:

```
class DoneAndNotDoneFutures(Sequence[set[Future[_T]]]):
    @property
    def done(self) -> set[Future[_T]]: ...
    @property
    def not_done(self) -> set[Future[_T]]: ...
    def __new__(_cls, done: set[Future[_T]], not_done: set[Future[_T]]) -> DoneAndNotDoneFutures[_T]: ...
    def __len__(self) -> int: ...
    @overload
    def __getitem__(self, __i: SupportsIndex) -> set[Future[_T]]: ...
    @overload
    def __getitem__(self, __s: slice) -> DoneAndNotDoneFutures[_T]: ...
```

Until two days ago, this stub actually had a bug: `done` and `not_done` were both given as writeable attributes, whereas they are read-only properties at runtime.

With generic NamedTuples, we could write the stub for the class far more simply (and more accurately) like this:

```
class DoneAndNotDoneFutures(NamedTuple, Generic[_T]):
    done: set[Future[_T]]
    not_done: set[Future[_T]]
```

And in code that actually needs to run at runtime, I frequently find it frustrating that I have to use dataclasses instead of NamedTuples if I want a simple class that just happens to be generic. dataclasses are great, but for small, lightweight classes, I prefer to use NamedTuples where possible. I often find that I don't need to use the full range of features dataclasses provide; and NamedTuples are often more performant than dataclasses, especially in cases where there's a lot of tuple unpacking.
msg414589 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-05 17:33
Okay, that's a sensible use case.

I do doubt your intuition of preferring named tuples over dataclasses a bit. This seems to encourage premature optimization. I'd say for simple cases use plain tuples (most performant), for complex cases use dataclasses (named fields and many other features that you may eventually want).

Compare concurent.futures.wait()'s return type (a named tuple) to asyncio.tasks.wait()'s return type (a plain tuple). I don't think that naming the fields of the return tuple (awkwardly :-) makes the c.f.wait() API easier to understand than the asyncio.wait() API.

Maybe named tuples, like typed dicts, are "in-between" solutions on the spectrum of data types (tuple - named tuple - dataclass; dict - typed dict - dataclass), and we should encourage people to use the neighboring solutions instead.

I'd rather spend efforts making dataclasses faster than adding features to named tuples.
msg414592 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-05 18:04
I sense we'll have to agree to disagree on the usefulness of NamedTuples in the age of dataclasses :)

For me, I find the simplicity of the underlying idea behind namedtuples — "tuples with some properties bolted on" — very attractive. Yes, standard tuples are more performant, but it's great to have a tool in the arsenal that's essentially the same as a tuple (and is backwards-compatible with a tuple, for APIs that require a tuple), but can also, like dataclasses, be self-documenting. (You're right that DoneAndNotDoneFutures isn't a great example of this.)

But I agree that this shouldn't be a priority if it's hard to accomplish; and there'll certainly be no complaints from me if energy is invested into making dataclasses faster.
msg414593 - (view) Author: Thomas Grainger (graingert) * Date: 2022-03-05 18:41
The main advantage for my usecase is support for heterogeneous unpacking

On Sat, Mar 5, 2022, 6:04 PM Alex Waygood <report@bugs.python.org> wrote:

>
> Alex Waygood <Alex.Waygood@Gmail.com> added the comment:
>
> I sense we'll have to agree to disagree on the usefulness of NamedTuples
> in the age of dataclasses :)
>
> For me, I find the simplicity of the underlying idea behind namedtuples —
> "tuples with some properties bolted on" — very attractive. Yes, standard
> tuples are more performant, but it's great to have a tool in the arsenal
> that's essentially the same as a tuple (and is backwards-compatible with a
> tuple, for APIs that require a tuple), but can also, like dataclasses, be
> self-documenting. (You're right that DoneAndNotDoneFutures isn't a great
> example of this.)
>
> But I agree that this shouldn't be a priority if it's hard to accomplish;
> and there'll certainly be no complaints from me if energy is invested into
> making dataclasses faster.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue43923>
> _______________________________________
>
msg414705 - (view) Author: Steven Silvester (Steven Silvester) Date: 2022-03-07 22:41
The use case that prompted https://github.com/python/cpython/pull/31679 is that we are adding typings to `PyMongo`.  We are late to using typings,  because we only recently dropped Python 2.7 support.  

We have an existing options class that subclasses `NamedTuple`.  We would like to make that class `Generic`, but are currently blocked.  

Our current workaround is to create a separate stub file that uses `class CodecOptions(Tuple, Generic[T])` and explicitly re-declares the `NamedTuple` API.

Switching to `dataclass` would be disruptive, since we still support Python 3.6 and only rely on the standard library.  We would also require a major version update since it would be an API change.
msg414709 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2022-03-08 00:37
Playing tricks where compile-time and run-time see slightly different types is probably more productive than trying to revert a PR that was in Python 3.9 and 3.10. :-)

I'm not opposed to supporting generic NamedTuple, but I expect the fix will never hit 3.9 and 3.10, and it needs to be a "fix forward" PR.

Would you mind closing the "revert" PR unmerged?
msg414754 - (view) Author: Steven Silvester (Steven Silvester) Date: 2022-03-08 14:02
I agree we're stuck with the typing stub workaround for our use case.  We can re-submit a "fix forward" PR.
msg414791 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-09 10:21
PR 31781 is a simple PR which enables multiple inheritance with NamedTuple. As a side effect, it adds support of generic NamedTuple.

I am not sure that all details work as expected. It is easy to limit multiple inheritance only for Generic if needed.
msg414792 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-09 10:35
+1 for the more minimal changeset proposed in PR 31781. I've never felt a need for NamedTuple multiple inheritance other than with Generic, so wouldn't be opposed to restricting it only to Generic.
msg414815 - (view) Author: Julius Park (juliusgeo) * Date: 2022-03-09 21:20
What about Protocol? It is possible to create a dataclass that is a protocol, so it would be nicer from a symmetry perspective to allow it on both dataclasses and NamedTuples.
msg414816 - (view) Author: Jelle Zijlstra (JelleZijlstra) * (Python committer) Date: 2022-03-09 21:27
A NamedTuple that is also a Protocol doesn't make sense to me, since a NamedTuple is a concrete (nominal) type and a Protocol cannot inherit from a concrete type. If you want something like that to happen, it's better to open an issue on https://github.com/python/typing first, so this issue can stay focused on support for Generic + NamedTuple.
msg414817 - (view) Author: Alex Waygood (AlexWaygood) * (Python triager) Date: 2022-03-09 21:53
I agree with Jelle — a valid protocol cannot inherit from a concrete type, and the whole point of NamedTuple is that it creates a tuple subclass (and tuple is obviously a concrete type).
msg414830 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2022-03-10 08:17
What about mix-ins or abstract base classes?

class VecMixin:
    def length(self):
        return math.hypot(*self)

class Vec2D(NamedTuple, VecMixin):
    x: float
    y: float

class Vec3D(NamedTuple, VecMixin):
    x: float
    y: float
    z: float

Currently you need to use the following trick to get a similar result:

class Vec2D(NamedTuple):
    x: float
    y: float

class Vec2D(Vec2D, VecMixin):
    pass
History
Date User Action Args
2022-04-11 14:59:44adminsetgithub: 88089
2022-03-10 21:38:13behackettsetnosy: + behackett
2022-03-10 08:17:42serhiy.storchakasetmessages: + msg414830
2022-03-09 21:53:10AlexWaygoodsetmessages: + msg414817
2022-03-09 21:27:06JelleZijlstrasetmessages: + msg414816
2022-03-09 21:20:50juliusgeosetmessages: + msg414815
2022-03-09 10:35:58AlexWaygoodsetmessages: + msg414792
2022-03-09 10:21:44serhiy.storchakasetmessages: + msg414791
2022-03-09 10:16:07serhiy.storchakasetpull_requests: + pull_request29887
2022-03-09 04:16:10juliusgeosetnosy: + juliusgeo
pull_requests: + pull_request29884
2022-03-08 14:02:35Steven Silvestersetmessages: + msg414754
2022-03-08 00:37:42gvanrossumsetmessages: + msg414709
2022-03-07 22:41:40Steven Silvestersetnosy: + Steven Silvester
messages: + msg414705
2022-03-05 18:41:34graingertsetmessages: + msg414593
2022-03-05 18:04:48AlexWaygoodsetmessages: + msg414592
2022-03-05 17:33:51gvanrossumsetmessages: + msg414589
2022-03-05 16:51:41AlexWaygoodsetnosy: + graingert
2022-03-05 16:49:09AlexWaygoodsetmessages: + msg414585
2022-03-05 16:08:04gvanrossumsetmessages: + msg414584
2022-03-05 09:06:46serhiy.storchakasetmessages: + msg414570
2022-03-05 08:28:20AlexWaygoodsetmessages: + msg414568
2022-03-05 08:26:53AlexWaygoodsetmessages: + msg414567
2022-03-05 05:12:39gvanrossumsetmessages: + msg414563
2022-03-05 02:57:54JelleZijlstrasetmessages: + msg414559
2022-03-05 02:55:26gvanrossumsetmessages: + msg414558
2022-03-05 01:40:45JelleZijlstrasetmessages: + msg414557
2022-03-04 17:54:39gvanrossumsetmessages: + msg414541
2022-03-04 10:20:15AlexWaygoodsetnosy: + gvanrossum, JelleZijlstra, sobolevn, kj
2022-03-04 09:40:03python-devsetkeywords: + patch
nosy: + python-dev

pull_requests: + pull_request29798
stage: patch review
2021-11-10 23:15:18AlexWaygoodsetnosy: + AlexWaygood
2021-11-10 18:57:26rhettingersetmessages: + msg406124
2021-11-10 11:48:28dlukessetnosy: + dlukes
messages: + msg406083
2021-04-23 19:12:50eric.smithsetnosy: + eric.smith
2021-04-23 16:27:00rhettingersetnosy: + rhettinger, serhiy.storchaka, levkivskyi
2021-04-23 16:23:07FHTMitchellsettype: behavior
2021-04-23 16:22:56FHTMitchellcreate