This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: simple "sequence" class ignoring __len__
Type: behavior Stage: resolved
Components: Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Eric.Wieser, josh.r, prudvinit, rhettinger, treddy
Priority: normal Keywords:

Created on 2018-08-25 05:26 by treddy, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Messages (5)
msg324037 - (view) Author: Tyler Reddy (treddy) * Date: 2018-08-25 05:26
Downstream in NumPy we've noticed that a "sequence" object defined as below will hang (infinite __getitem__ calls) if we try to turn it into an array. The same holds in CPython for converting it to a list:

class OneList:

    def __len__(self):
        # this won't be checked by
        # PySequence_Fast and several
        # over C API calls
        return 1

    def __getitem__(self, x):
        # called indefinitely by
        # i.e., PySequence_Fast
        return 1

Just to confirm -- this is intentional / desired behavior:

list(OneList()) should hang in CPython?

related: 
https://github.com/numpy/numpy/issues/8912
https://github.com/numpy/numpy/pull/11815
https://stackoverflow.com/a/43566241/2942522
msg324056 - (view) Author: Prudvi RajKumar Maddala (prudvinit) * Date: 2018-08-25 09:13
I think it should hang, since we are not throwing any StopIteration exception to break the infinite loop
msg324083 - (view) Author: Eric Wieser (Eric.Wieser) * Date: 2018-08-25 17:31
What I think I find surprising is that I'd expect the sequence protocol to be defined by `__getitem__` and `__len__`, and for `__iter__` to be inferred as:

    def __iter__(self):
        for i in range(len(self)):
            yield self[i]

But in reality it seems it is inferred only from `__getitem__`, as:

    def __iter__(self):
        i = 0
        while True:
            try:
                yield self[i]
            except IndexError:
                return
            i += 1
msg324119 - (view) Author: Josh Rosenberg (josh.r) * (Python triager) Date: 2018-08-26 14:43
That's the documented behavior. Per https://docs.python.org/3/reference/datamodel.html#object.__getitem__ :

>Note: for loops expect that an IndexError will be raised for illegal indexes to allow proper detection of the end of the sequence. 

The need for *only* __getitem__ is also mentioned in the documentation of the iter builtin ( https://docs.python.org/3/library/functions.html#iter ):

>Without a second argument, object must be a collection object which supports the iteration protocol (the __iter__() method), or it must support the sequence protocol (the __getitem__() method with integer arguments starting at 0).

At no point is a dependency on __len__ mentioned.
msg324127 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2018-08-26 17:24
I concur with Josh.  This matches the documented behavior and isn't a bug. Marking as closed.

There are potentially two ways to stop sequence iteration, either by using len or by waiting for IndexError.  Python uses the latter to allow lists to be dynamically resized during iteration and because checking len on every iteration would be expensive.  

If someone doesn't read the docs, and assumes Python uses len, and writes an inconsistent class (one where len doesn't match the point where IndexError is raised), then there is little we can do to prevent that person from being "surprised".
History
Date User Action Args
2022-04-11 14:59:05adminsetgithub: 78675
2018-08-26 17:24:49rhettingersetstatus: open -> closed
resolution: not a bug
messages: + msg324127

stage: resolved
2018-08-26 14:43:43josh.rsetnosy: + josh.r
messages: + msg324119
2018-08-25 20:01:12rhettingersetassignee: rhettinger

nosy: + rhettinger
2018-08-25 17:31:35Eric.Wiesersetmessages: + msg324083
2018-08-25 17:00:25Eric.Wiesersetnosy: + Eric.Wieser
2018-08-25 09:13:51prudvinitsetnosy: + prudvinit
messages: + msg324056
2018-08-25 05:26:30treddycreate