Title: Incorrect description of descriptor invocation in Python Language Reference
Type: enhancement Stage:
Components: Documentation Versions: Python 3.4, Python 3.5, Python 2.7
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Justin.Eldridge, docs@python, r.david.murray
Priority: normal Keywords:

Created on 2015-01-25 17:16 by Justin.Eldridge, last changed 2015-01-26 17:54 by Justin.Eldridge.

Messages (3)
msg234676 - (view) Author: Justin Eldridge (Justin.Eldridge) Date: 2015-01-25 17:16
The section titled "Invoking Descriptors" in the Python Language Reference [1]

    Class Binding
        If binding to a new-style class, A.x is transformed into the call: 
        A.__dict__['x'].__get__(None, A).

This suggests that __get__ is looked up on the instance of x, when I believe
that it is actually looked up on the type. That is, it's my understanding that
A.x invokes:

    type(A.__dict__['x']).__get__(A.__dict__['x'], None, Foo))

Here's some Python 3.4 code demonstrating this:

    class A:

    class Descriptor:
        def __get__(self, obj, cls):
    A.x = Descriptor()

    def replacement_get(obj, cls):
        print("This is a replacement.")
    A.__dict__['x'].__get__ = replacement_get

Now, writing:

    >>> A.x

    >>> A.__dict__['x'].__get__(None, A)
    This is a replacement!

    >>> type(A.__dict__['x']).__get__(A.__dict__['x'], None, A)

The documentation makes a similar statement about instance binding that also
appears to be incorrect. This is the case in all versions of the document I
could find.

What I believe to be the actual behavior is implied by a later section in the
same document, which states that the implicit invocation of special methods is
only guaranteed to work correctly if the special method is defined on the type,
not the instance.  This suggests that the statements in "Invoking Descriptors"
aren't quite correct, and while the true behavior is a little more verbose, I
think it would be worthwhile to update the documentation so as to avoid

msg234684 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-01-25 20:28
I believe that you are correct: special methods are looked up on the type, and this is assumed implicitly in the Class Binding description. (This was not 100% true in python2, but it is in current python3.)  But the Class Binding description is correct, since if the __get__ method is *not* defined on the type (of the descriptor), the descriptor instance itself is returned (so explicitly calling type in the "equivalent expression" would be wrong).

This is one of the most complex topics to describe in Python (I still don't have it solid in my head and I've been working with Python for years now).  If we can come up with clearer wording that is good, but we've tried several times already to improve it :(

I don't know what you are referring to for the 'instance binding' that makes the same 'mistake', but I suspect it is also covered by the "special methods are looked up on the type" rule.
msg234763 - (view) Author: Justin Eldridge (Justin.Eldridge) Date: 2015-01-26 17:54
Ah, I see how writing a description of this which is both concise and precise
would be difficult, especially for Python 2.

> But the Class Binding description is correct, since if the __get__ method is
> *not* defined on the type (of the descriptor), the descriptor instance itself
> is returned (so explicitly calling type in the "equivalent expression" would be
> wrong)

I see, but couldn't this also be held against the current "equivalent"?  That
is, saying A.x is equivalent to A.__dict__['x'].__get__(None, A) is not stricly
true when __get__ isn't defined on type(x).

I think I see now why this is difficult to document cleanly, and the "Special
method lookup" section of the documentation does a good job of explaining this.
The issue isn't exclusive to descriptors. It affects, for example, the
documentation on rich comparison operators, which says that x == y invokes
x.__eq__(y), when this hasn't quite been true since old-style classes.

So saying x == y is equivalent to x.__eq__(y) isn't really correct, and saying
that it is equivalent to type(x).__eq__(x,y) isn't quite right either, as
implicit invocation may bypass the metaclass's __getattribute__. The latter,
however, seems "less wrong". Is there a reason that the former is preferred by
the documentation?
Date User Action Args
2015-01-26 17:54:08Justin.Eldridgesetmessages: + msg234763
2015-01-25 20:28:10r.david.murraysetnosy: + r.david.murray

messages: + msg234684
versions: - Python 3.2, Python 3.3, Python 3.6
2015-01-25 17:16:03Justin.Eldridgecreate