classification
Title: Inhomogeneous behaviour for descriptors in between the class-instance and metaclass-class pairs
Type: behavior Stage: resolved
Components: Documentation Versions: Python 3.8
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Hugo Ricateau, eric.snow, rhettinger
Priority: normal Keywords:

Created on 2020-01-24 15:17 by Hugo Ricateau, last changed 2020-01-26 06:04 by rhettinger. This issue is now closed.

Messages (5)
msg360623 - (view) Author: Hugo Ricateau (Hugo Ricateau) Date: 2020-01-24 15:17
Assume one has defined the following descriptor:
```
class Descriptor:
    def __set__(self, instance, value):
        print('SET')
```

On the one hand, for the class-instance pair, the behaviour is as follows:
```
class FirstClass:
    descriptor = Descriptor()

    def __init__(self):
        self.descriptor = None

FirstClass().descriptor = None
```
results in "SET" being displayed twice; i.e. both assignations triggered the __set__ method of the descriptor.

On the other hand, for the metaclass-class pair, the behaviour is the following:
```
class SecondClassMeta(type):
    descriptor = Descriptor()

class SecondClass(metaclass=SecondClassMeta):
    descriptor = None

SecondClass.descriptor = None
```
results in "SET" being displayed only once: the first assignation (the one in the class definition) did not triggered __set__.

It looks to me like an undesirable asymmetry between the descriptors behaviour when in classes vs when in metaclasses. Is that intended? If it is, I think it should be highlighted in the descriptors documentation.

Best
msg360644 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-01-24 19:36
First of all, thanks for asking about this.  Everything is working as expected.  Let's look at why.

First, be sure the behavior of descriptors is clear: the descriptor protocol is only triggered by "dotted access" on an object ("obj.attr").  So you should expect it only where you see that syntax used.

Let's look at your examples now.

> FirstClass().descriptor = None

In this case there are 2 dotted accesses.  The first one happens in __init__() when the object is created.  The second is the rest of the above line.

> class SecondClass(metaclass=SecondClassMeta):
>    descriptor = None
>
> SecondClass.descriptor = None

In this case there is only one dotted access, in that last line.  The object in this case is SecondClass and its class is SecondClassMeta.  Unlike with FirstClass, the *class* in the second example (SecondClassMeta) does not have a __init__() with the dotted access.  Instead there is only the one dotted access afterward.  If SecondClassMeta had the same __init__() that FirstClass had then you would have seen a second trigger of the descriptor.

It seems you expected assignments (name binding) in the class definition body to be treated the same as dotted access.  They are not.  This is because when a class definition body is evaluated, the class object does not exist yet.  The steps for class creation go like this:

1. figure out the metaclass (by default "type")
2. calls its __prepare__() method to get a namespace
3. execute the class body (like a function) with that namespace as the locals
4. create the class object, passing in that namespace

Python has worked this way since version 2.2 (PEP 252).  See: https://docs.python.org/3/reference/datamodel.html#creating-the-class-object

If you want to get clever you could return a namespace object from your metaclass __prepare__ that triggers the descriptor protocol as you expected.  However, I would not recommend that.  Getting clever with metaclasses is best avoided.  The default behavior is much simpler.  That won't be changing.

> It looks to me like an undesirable asymmetry between the descriptors behaviour when in classes vs when in metaclasses. Is that intended? If it is, I think it should be highlighted in the descriptors documentation.

Regardless, metaclasses are used infrequently and combining them with descriptors (especially relative to class definitions) is even less common.  So pointing out the caveats of this case may not be worth the time of all future readers of those docs.

That said, clearly it would have helped you in this case. :)  So here are some *possible* places to clarify (very briefly):

* descriptors howto
   + about mixing descriptors with metaclasses
   + a list enumerating places where descriptors are *not* invoked
* language reference (metaclasses section)
   + a warning saying something like "Avoid metaclasses if you can help it and only use them if you have a clear understanding of Python's object model (and dotted access)"
* language reference (descriptors/dotted access section)
   + a list enumerating places where descriptors are *not* invoked

Which of those do you think would have helped you the most?
msg360645 - (view) Author: Eric Snow (eric.snow) * (Python committer) Date: 2020-01-24 19:37
@Raymond, What do you think about adding a helpful note or two in the docs?
msg360685 - (view) Author: Hugo Ricateau (Hugo Ricateau) Date: 2020-01-25 14:05
Thanks for this detailed answer; very instructive :)

> the descriptor protocol is only triggered by "dotted access"

Indeed; this is what I was missing... despite it is indirectly mentioned in the documentation. Nonetheless, it could be worth the overload to explicitly add in the language reference that 'the descriptor protocol is only triggered by "dotted access"' (looks like it is not the case for now).

>    + a list enumerating places where descriptors are *not* invoked
> [...]
> Which of those do you think would have helped you the most?

Could be really helpful as well, by clearly exhibiting the limitations of the descriptors; I think the best location for this could be the 'descriptors howto' page despite the other option is perfectly suitable as well.

Best,
Hugo
msg360720 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2020-01-26 06:04
Some thoughts:

* The docs talk about descriptor invocation from "attribute access".  The reason they don't say "dotted access" is that the descriptors can be invoked in multiple ways: dotted access, getattr()/setattr() functions, super(), or direct calls to __getattribute__().

* From the point-of-view of descriptors, metaclasses aren't special.  The only essential fact needed is that type.__getattribute__() is called instead of object.__getattribute__().  The how-to guide already discusses how those methods differ.

* For now, I'm inclined to leave the docs as-is.  The existing coverage of common cases is already a bit hard to read.  It could become less readable if we list places where something doesn't happen, warnings to avoid features, or detailed explanations of uncommon cases like mixing metaclasses with descriptors.  If this arises again, we could add a FAQ entry of some such.
History
Date User Action Args
2020-01-26 06:04:03rhettingersetstatus: open -> closed
resolution: not a bug
messages: + msg360720

stage: resolved
2020-01-25 20:56:29rhettingersetassignee: rhettinger
components: + Documentation, - Interpreter Core
versions: - Python 3.6, Python 3.7
2020-01-25 14:05:57Hugo Ricateausetmessages: + msg360685
2020-01-24 19:37:28eric.snowsetnosy: + rhettinger
messages: + msg360645
2020-01-24 19:36:07eric.snowsetnosy: + eric.snow
messages: + msg360644
2020-01-24 15:17:34Hugo Ricateaucreate