Title: The 'in' syntax should work with any object that implements __iter__
Type: behavior Stage: resolved
Components: Interpreter Core Versions: Python 3.6
Status: closed Resolution: rejected
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Edouard KLEIN, rhettinger, steven.daprano, xiang.zhang
Priority: normal Keywords:

Created on 2017-05-12 12:18 by Edouard KLEIN, last changed 2017-05-12 21:41 by rhettinger. This issue is now closed.

Messages (4)
msg293547 - (view) Author: Edouard KLEIN (Edouard KLEIN) Date: 2017-05-12 12:18
In this StackOverflow question:

the question of why an object that implements __iter__ through __getattr__ does not work with the "in" syntax is raised. The accepted answer is that the interpreter checks for __iter__ instead of just calling it.

If that's the case, I think that:
- Either the behaviour of the interpreter should be changed to accept any object that implements __iter__, regardless of how,
- Or, if there is a technical reason why the interpreter can't change its behavior, the documentation should make it clear that __iter__ must be hard coded into the object.

Here is the code of the object that arguably should be iterable but is not:

class IterOrNotIter:
    def __init__(self):
        self.f = open('/tmp/toto.txt')
    def __getattr__(self, item):
            return self.__getattribute__(item)
        except AttributeError:
            return self.f.__getattribute__(item)

IterOrNotIter().__iter__().__next__()  # Works
'a' in IterOrNotIter()  # Raises a TypeError
msg293548 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2017-05-12 13:29
This is not a small change, and will need careful thought.

The problem is that in Python 3 (and in Python 2 for new-style classes), dunder methods are only called by the interpreter if they are defined on the class itself, not on the instance. That's an optimization, and it does mean that (for example) automatic delegation doesn't work with dunder methods.

But if we change this, we should change it for *all* dunder methods, not just __iter__. It would be a problem if we start "fixing" dunders piecemeal, one at a time. Either they should all work (like with classic classes) or none of them should.

So this is potentially a big change.
msg293551 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2017-05-12 14:39
A further thought... looking at your example code, I believe that part of the __getattr__ is redundant.

    def __getattr__(self, item):
            return self.__getattribute__(item)
        except AttributeError:
            return self.f.__getattribute__(item)

__getattr__ is only called if normal attribute lookup has already failed, so the call to self.__getattribute__ is unnecessary. If it would have succeeded, it would have already succeeded and __getattr__ won't have been called at all.

For more discussion on how to do automatic delegation, you should look at Alex Martelli's recipe from the Python Cookbook:

Also, its a bit... funny... to call dunder methods directly. (I'm deliberately not using the word "wrong".) They are implementation, not interface. I think your __getattr__ should be:

    def __getattr__(self, name):
        return getattr(self.f, name)

It also looks nicer :-)
msg293576 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-05-12 21:41
-1 on this proposal.  We haven't really seen a need for this in practice.  And such as change would be likely create unforeseen consequences to existing code.  

The proposal is at odds with the existing design.  Python internals work by checking for whether a given slot is filled-out.  Trying to make these lookups transparent to __getattr__ calls would be tricky at best (we can't know what __getattr__ does without calling it) and definitely not fast.

Even if it could be done easily, I would still prefer the current design which requires explicit delegation of magic methods by proxy objects.

FWIW, me made the same decision for super() objects ( ):

Note that super() is implemented as part of the binding process for explicit dotted attribute lookups such as super().__getitem__(name). It does so by implementing its own __getattribute__() method for searching classes in a predictable order that supports cooperative multiple inheritance. Accordingly, super() is undefined for implicit lookups using statements or operators such as super()[name].

For the reasons listed above and those listed by Steven, I'm marking this as closed/rejected.  Thank you for the suggestion, but we need to decline.
Date User Action Args
2017-05-12 21:41:31rhettingersetstatus: open -> closed
messages: + msg293576

assignee: rhettinger
resolution: rejected
stage: resolved
2017-05-12 14:39:13steven.dapranosetmessages: + msg293551
2017-05-12 13:59:57xiang.zhangsetnosy: + rhettinger, xiang.zhang
2017-05-12 13:29:36steven.dapranosetnosy: + steven.daprano
messages: + msg293548
2017-05-12 12:18:04Edouard KLEINcreate