classification
Title: collections.abc.Indexable
Type: enhancement Stage: patch review
Components: Library (Lib) Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: levkivskyi Nosy List: Tim.Graham, abarnert, gvanrossum, levkivskyi, lukasz.langa, ned.deily, rhettinger, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2016-01-01 22:18 by abarnert, last changed 2018-02-18 18:18 by ned.deily.

Pull Requests
URL Status Linked Edit
PR 5414 merged rhettinger, 2018-01-29 07:53
PR 5460 merged serhiy.storchaka, 2018-01-31 14:42
PR 5734 merged levkivskyi, 2018-02-18 13:38
PR 5743 merged ned.deily, 2018-02-18 18:08
Messages (26)
msg257311 - (view) Author: Andrew Barnert (abarnert) * Date: 2016-01-01 22:18
In an -ideas thread, Guido suggested (http://article.gmane.org/gmane.comp.python.ideas/37599):

> If we want some way to turn something that just defines __getitem__ and __len__ into a proper sequence, it should just be made to inherit from Sequence, which supplies the default __iter__ and __reversed__. (Registration is *not* good enough here.) If we really want a way to turn something that just supports __getitem__ into an Iterable maybe we can provide an additional ABC for that purpose; let's call it a HalfSequence until we've come up with a better name. (We can't use Iterable for this because Iterable should not reference __getitem__.)

Later in the thread, Nick Coghlan suggested (http://article.gmane.org/gmane.comp.python.ideas/37614):

> Perhaps collections.abc.Indexable would work? Invariant:

>     for idx, val in enumerate(container):
>         assert container[idx] is val

> That is, while enumerate() accepts any iterable, Indexable containers
have the additional property that the contained values can be looked
up by their enumeration index. Mappings (even ordered ones) don't
qualify, since they offer a key:value lookup, but enumerating them
produces an index:key relationship.

So, in particular:

* Indexable is a subclass of Iterable.
* Sequence is a subclass of Indexable, Sized, and Container instead of Iterable, Sized, and Container. (Or, if #25987 also goes through, of Reversible, Indexable, Sized, and Container.)
* The abstract method __getitem__ and the concrete __iter__ implementation get moved up from Sequence to Indexable.
* Indexable does _not_ have a subclass hook (to avoid making every Mapping, generic type, etc. accidentally Indexable).

This means that you can write this (borrowing an example from Steven D'Aprano in http://article.gmane.org/gmane.comp.python.ideas/23369/):

    class Squares(collections.abc.Indexable):
        def __getitem__(self, index):
            return index**2

Because this no longer depends on the old-style sequence protocol, testing it with ABCs will work as expected.

For related issues, see #25987, #25864, #25958, and https://github.com/ambv/typehinting/issues/170
msg257487 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2016-01-04 20:30
Interesting. I just hope the name doesn't confuse people into connecting this with the Index type (a type of integer that can't be cast from float).
msg298712 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-07-20 10:07
The last few weeks something bothered while working on Protocols PEP: protocols should be ideally compact (and PEP already emphasizes this).
However, the only potential candidates for __getitem__ are Sequence and Mapping, that are both quite bulky (half dozen members each). I was thinking about different options like adding BaseMap as a base for Mapping and Sequence that will only have __getitem__. I expect this to be a popular protocol, since often people just need something that can be subscripted.

Fortunately I stumbled into this issue. It looks like the optimal way now is:

* Have an abstract base class (let's call it BaseMap, although I don't really like this name) in collections.abc that has only __getitem__ method.
* It will be inherited by both Sequence and Mapping, but for the purpose of static typing, Sequence[T] will be a subtype of BaseMap[int, T], while Mapping[KT, VT] will be a subtype of BaseMap[KT, VT].
* BaseMap will be contravariant in key, this will solve problems with Mapping (invariant in key), for example a function that expects BaseMap[str, int] will accept Dict[Union[str, unicode], int].
* BaseMap will be a protocol in typing, so that people can extend it depending on their needs (e.g. add a .get() method).

Guido, Łukasz if you agree, then I will add this to the Protocols PEP and  will make a PR to collections.abc. We need to agree on the name, there are two options now: BaseMap and Indexable. I don't like the second since I would rather have it as a generic alias: Indexable = BaseMap[int, T]. However, BaseMap is also not very good, since I want something more "neutral" between Mapping and Sequence.
msg298728 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-07-20 15:29
IIRC, the original motivation for ABCs was to differentiate distinct uses of __getitem__ (we forever struggled with differentiating sequences from mapping).  It seems to me that this proposal is a step backwards.  Other than a feeling of lightness, I don't think this proposal does anything for us.  What is point of knowing an object is Subscriptable without knowing how it is to be used.

The OP has a sense that Mapping and Sequence are "too heavy" but I think the reality that useful classes almost never use __getitem__ in isolation; rather, it is part of a small constellation of methods that are typically used together.  I would prefer that collections.abc continue to reflect that reality.

Also, I worry that collections.abc is becoming cluttered.  The existence of use ABCs like MutableMapping is being drowned-out by one-trick-ponies.  We're developing an unfavorable ratio of theoretical building blocks versus the practical tools.
msg298734 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-07-20 16:50
Raymond,

> The existence of use ABCs like MutableMapping is being drowned-out by one-trick-ponies.  We're developing an unfavorable ratio of theoretical building blocks versus the practical tools.

Why do you think they are "theoretical"? FWIW, a simple search on GitHub for abc.X gives this:
Sequence 322K
Mapping 267K
Iterable 244K
Container 99K
MutableMapping 77K

I don't see any pattern here (although this may be only anecdotal evidence :-)

> Other than a feeling of lightness, I don't think this proposal does anything for us.

This proposal makes more sense (and motivation) in the context of static typing, since it is a good way to know how __getitem__ is going to be used -- from its static type. Something typed with Subscriptable[int, T] would accept both Sequence[T] and Mapping[int, T]. The latter two are  currently differentiated by __reversed__ (Sequence is Reversible as opposed to Mapping).

Concerning an invariant that order of iteration is consistent with indexing by subsequent integers, I think this can't be checked reliably neither statically, nor by any simple runtime isinstance check. For example a subclass can override Indexable.__iter__ and break the iteration order. We can still add it, and rely on user cooperation.

Anyway, I am not insisting on adding either Subscriptable nor Indexable, but I think it is important that these things are discussed in view of PEP 544.

Alternative proposal would be to add Subscriptable and its subclass Indexable only to typing as protocols (with some special-casing in type checkers for the latter to provide a higher level of contract).
(Concerning the name, I think Enumerable sounds better than Indexable, IIUC the idea is to work correctly with enumerate.)
msg298751 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-07-20 21:38
At least one use of Indexable (by whatever name, but this rolls off the tongue the easiest) would be the metaclass in typing.py that allows one to write List[int] -- there's no containter here!

The "drowning out" seems purely subjective -- perhaps if you order the contents of the module alphabetically you won't get much insight, but the docs should use a better organizing principle than that. (After all alphabetization was invented as a crutch for searching. :-)
msg298777 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-07-21 07:23
Guido, do you think there is some way to decouple collections.abc from spilling into the namespace for non-abc collections module?

At the interactive prompt, running dir() on collections gives an alphabetical hodgepodge of the two modules.  To my eyes, it is difficult to discern the concrete from the abstract.

>>> dir(collections)
['AsyncGenerator', 'AsyncIterable', 'AsyncIterator', 'Awaitable', 'ByteString', 'Callable', 'ChainMap', 'Collection', 'Container', 'Coroutine', 'Counter', 'Generator', 'Hashable', 'ItemsView', 'Iterable', 'Iterator', 'KeysView', 'Mapping', 'MappingView', 'MutableMapping', 'MutableSequence', 'MutableSet', 'OrderedDict', 'Reversible', 'Sequence', 'Set', 'Sized', 'UserDict', 'UserList', 'UserString', 'ValuesView', '_Link', '_OrderedDictItemsView', '_OrderedDictKeysView', '_OrderedDictValuesView', '__all__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '_chain', '_class_template', '_collections_abc', '_count_elements', '_eq', '_field_template', '_heapq', '_iskeyword', '_itemgetter', '_proxy', '_recursive_repr', '_repeat', '_repr_template', '_starmap', '_sys', 'abc', 'defaultdict', 'deque', 'namedtuple']
msg298790 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2017-07-21 10:03
Names from collections.abc are re-exported to collections for backward compatibility. IIRC Serhiy also wanted to stop re-exporting them. I am not sure whether we need any "deprecation period" for this.
msg298791 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-21 10:10
I'm not sure that *new* names in collections.abc should be re-exported to collections. This isn't required for backward compatibility.
msg298810 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-07-21 15:57
Yes we will need a deprecation period for this starting with 3.7. It's fine
not to import names newly added to collections.abc in 3.7 (i.e. collections
would have to explicitly import everything that it got via * in 3.6).
msg298830 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-07-22 02:20
> Yes we will need a deprecation period for this starting with 3.7.

There are two ways to do this.  A wimpy and clean way is to just add a deprecation notice in the docs.  The thorough and messy way is to temporarily copy everything from the 3.6 collections.abc into the regular collections module and then insert deprecation warning code directly in each of copied classes.
msg298835 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-07-22 04:49
The copying approach likely won't work because there will be code that uses isinstance(x, collections.abc.Sequence) and other code that uses isinstance(x, collections.Sequence) and they'll expect both to be true.
msg298838 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2017-07-22 07:01
If we want to add programming warnings, we should replace sys.modules['collections'] with an instance of a module type subclass with corresponding properties raising a DeprecationWarning.
msg298879 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-07-22 23:59
> The copying approach likely won't work ...

Then it seems like just adding a note to the docs is the way to go. That's probably okay though.  I don't think we've every gotten much benefit from the programmatic deprecation warnings.
msg298887 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2017-07-23 04:12
I think it's not worth the hack that Serhiy suggests. But we should make
some noise around this change in the 3.7 announcement.
msg311147 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-01-29 16:27
New changeset e6d342156d2ab20fb88c0a5ec615fa8f602c0769 by Raymond Hettinger in branch 'master':
bpo-25988: Deprecate exposing collections.abc in collections GH-5414
https://github.com/python/cpython/commit/e6d342156d2ab20fb88c0a5ec615fa8f602c0769
msg311310 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-01-31 05:16
Guido:
> But we should make some noise around this change in the 3.7 announcement.

How about a paragraph in the Deprecation section of the "What's New in Python 3.7"? document

https://docs.python.org/dev/whatsnew/3.7.html#deprecated
msg311311 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-01-31 05:48
> > But we should make some noise around this change in the 3.7 announcement.
>
> How about a paragraph in the Deprecation section of the "What's New in Python 3.7"? document
>
> https://docs.python.org/dev/whatsnew/3.7.html#deprecated

You mean other than what's there? The text that's currently there begins with "In Python 3.8, ..." which almost sounds like it's misplaced or a typo.  Maybe you can amend it somehow to explain that in 3.7, using or importing the ABCs from "collections" instead of from "collections.abc" is deprecated, and in 3.8 it will stop working.
msg311319 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-31 10:12
Can't we use the new shiny PEP 562 here to actually emit a warning when collections.Iterable is used instead of collections.abc.Iterable?
msg311335 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-31 14:48
> Can't we use the new shiny PEP 562 here to actually emit a warning when collections.Iterable is used instead of collections.abc.Iterable?

PR 5460 implements this.

It exposes issues in third-party packages. urllib3 and pyparsing directly use or import ABCs from 'collections'. request and pip are indirectly affected. If not add these warnings in 3.7 we can break the world in 3.8.
msg311341 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2018-01-31 16:24
Good idea, Victor, and thanks for the PR, Serhiy!
msg311346 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-31 17:19
New changeset c66f9f8d3909f588c251957d499599a1680e2320 by Serhiy Storchaka in branch 'master':
bpo-25988: Emit a warning when use or import ABCs from 'collections'. (#5460)
https://github.com/python/cpython/commit/c66f9f8d3909f588c251957d499599a1680e2320
msg311668 - (view) Author: Tim Graham (Tim.Graham) * Date: 2018-02-05 14:33
The last commit that added the deprecation warning needs to be added to the 3.7 branch.
msg312316 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2018-02-18 17:41
New changeset 0442de5ad7835814d60f46c22a22942abb101aef by Ivan Levkivskyi in branch '3.7':
bpo-25988: Emit a warning when use or import ABCs from 'collections'. (GH-5734)
https://github.com/python/cpython/commit/0442de5ad7835814d60f46c22a22942abb101aef
msg312317 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-02-18 17:44
Thanks, Tim, for noticing that and thanks, Ivan, for taking care of it.  I should have cherrypicked this into 3.7.0b1.
msg312321 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2018-02-18 18:18
New changeset 2e84e47626c6eafacc9f011cd9fccc8bf1c8508e by Ned Deily in branch '3.7':
bpo-25988: add NEWS entry for 3.7.0b2 (#5743)
https://github.com/python/cpython/commit/2e84e47626c6eafacc9f011cd9fccc8bf1c8508e
History
Date User Action Args
2018-02-18 18:18:46ned.deilysetmessages: + msg312321
2018-02-18 18:08:55ned.deilysetpull_requests: + pull_request5520
2018-02-18 17:44:28ned.deilysetmessages: + msg312317
2018-02-18 17:41:01levkivskyisetmessages: + msg312316
2018-02-18 13:38:28levkivskyisetpull_requests: + pull_request5515
2018-02-05 14:33:43Tim.Grahamsetnosy: + Tim.Graham
messages: + msg311668
2018-01-31 17:19:37serhiy.storchakasetmessages: + msg311346
2018-01-31 16:24:14gvanrossumsetmessages: + msg311341
2018-01-31 14:48:22serhiy.storchakasetmessages: + msg311335
2018-01-31 14:42:43serhiy.storchakasetpull_requests: + pull_request5287
2018-01-31 10:12:52vstinnersetnosy: + vstinner
messages: + msg311319
2018-01-31 05:48:09gvanrossumsetmessages: + msg311311
2018-01-31 05:16:12ned.deilysetnosy: + ned.deily
messages: + msg311310
2018-01-29 16:27:53rhettingersetmessages: + msg311147
2018-01-29 07:53:31rhettingersetkeywords: + patch
stage: patch review
pull_requests: + pull_request5248
2017-07-23 04:12:51gvanrossumsetmessages: + msg298887
2017-07-22 23:59:17rhettingersetmessages: + msg298879
2017-07-22 07:01:30serhiy.storchakasetmessages: + msg298838
2017-07-22 04:49:46gvanrossumsetmessages: + msg298835
2017-07-22 02:20:57rhettingersetmessages: + msg298830
2017-07-21 15:57:09gvanrossumsetmessages: + msg298810
2017-07-21 10:10:19serhiy.storchakasetmessages: + msg298791
2017-07-21 10:03:43levkivskyisetnosy: + serhiy.storchaka
messages: + msg298790
2017-07-21 07:23:14rhettingersetmessages: + msg298777
2017-07-20 21:38:42gvanrossumsetmessages: + msg298751
2017-07-20 16:50:54levkivskyisetmessages: + msg298734
2017-07-20 15:29:47rhettingersetnosy: + rhettinger
messages: + msg298728
2017-07-20 10:07:35levkivskyisetversions: + Python 3.7
nosy: + lukasz.langa, levkivskyi

messages: + msg298712

assignee: levkivskyi
2016-01-04 20:30:05gvanrossumsetnosy: + gvanrossum
messages: + msg257487
2016-01-01 22:18:59abarnertcreate