classification
Title: Misleading documentation for NotImplemented
Type: enhancement Stage:
Components: Documentation Versions: Python 3.9, Python 3.8, Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: docs@python Nosy List: docs@python, murali, steven.daprano
Priority: normal Keywords:

Created on 2019-12-20 16:42 by murali, last changed 2019-12-28 07:11 by terry.reedy.

Messages (5)
msg358719 - (view) Author: Murali Ganapathy (murali) Date: 2019-12-20 16:42
The documentation at https://docs.python.org/3.6/library/constants.html#NotImplemented states

If all attempts return NotImplemented, the interpreter will raise an appropriate exception. However this is not true for __eq__.

===
class Foo:
  def __eq__(self, other):
    return NotImplemented

Foo() == Foo() # returns False, does not throw an exception
====
msg358722 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-20 17:39
This is because your class implicitly inherits from object and object.__eq__() is implemented and does not return NotImplemented.
msg358723 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2019-12-20 17:42
And to be more specific, == is guaranteed to work by falling back to object.__eq__() which falls back to object identity if the object doesn't have a custom __eq__() method.
msg358726 - (view) Author: Murali Ganapathy (murali) Date: 2019-12-20 18:40
===
# python3
class Base:
  def __eq__(self, other):
    print("base called")
    return super().__eq__(other)


class Foo(Base):

  def __eq__(self, other):
    print("foo called")
    return NotImplemented

Foo() == Foo()
# foo called
# foo called
False
====

Base.__eq__ is not called here. Is calling of object.__eq__ special cased?
msg358752 - (view) Author: Steven D'Aprano (steven.daprano) * (Python committer) Date: 2019-12-21 00:08
The behaviour is correct: `==` should not raise, even if both objects' `__eq__` method returns NotImplemented.

I agree that the documentation needs improvement. Looking at the current version:

https://docs.python.org/3/library/constants.html#NotImplemented

it says:

    Note: When a binary (or in-place) method returns NotImplemented 
    the interpreter will try the reflected operation on the other 
    type (or some other fallback, depending on the operator). If all
    attempts return NotImplemented, the interpreter will raise an
    appropriate exception. Incorrectly returning NotImplemented will
    result in a misleading error message or the NotImplemented value
    being returned to Python code.


(1) In the case of `==` and `!=`, an exception is not raised even all relevant methods return NotImplemented.

The source code for PyObject_RichCompare in object.c says:

    /* If neither object implements it, provide a sensible default
       for == and !=, but raise an exception for ordering. */

and then falls back to object identity.

If we wanted to lawyer up, the current docs don't say that "some other fallback" is limited to Python dunder methods. But that's the strong impression it gives, and it is actively misleading in that there's no hint that the fallback includes a default notion of equality as object identity, built right into the interpreter itself.

Even Brett got this wrong: object.__eq__ can return NotImplemented, so it isn't a suitable final fallback:

    py> object.__eq__(o, None)
    NotImplemented


so I think these docs could do with some improvement. Based on the current docs, I too would expect equality to raise an exception for a class that defines `__eq__`  and `__ne__` to always return NotImplemented.


(2) The last sentence, about "incorrectly returning NotImplemented", confuses me. Why are we warning about that? If you incorrectly return any value, whether it is NotImplemented, True, False, None or 3.1415, it will result in a misleading error message or incorrect value being returned.

Is it perhaps meant to say "incorrectly *raising* NotImplemented"?

I'm reopening this for a doc enhancement. Unfortunately I can't use github for technical reasons, so can't do a PR, but I'll suggest an updated description for the first part. (I have no idea what to do for the second part.)

    When a binary (or in-place) method returns NotImplemented 
    the interpreter will try the reflected operation on the other 
    type (or some other fallback, depending on the operator). If all
    attempts return NotImplemented, the interpreter will fall back
    to object identity for `==` and `!=` or raise an appropriate
    exception for all other comparisons.
History
Date User Action Args
2019-12-28 07:11:38terry.reedysetversions: - Python 3.5, Python 3.6
2019-12-24 17:54:06brett.cannonsetnosy: - brett.cannon
2019-12-21 00:09:14steven.dapranosettitle: Misleading documentation -> Misleading documentation for NotImplemented
2019-12-21 00:08:22steven.dapranosetstatus: closed -> open

type: enhancement

nosy: + steven.daprano
messages: + msg358752
resolution: not a bug ->
stage: resolved ->
2019-12-20 18:40:55muralisetmessages: + msg358726
2019-12-20 17:42:00brett.cannonsetmessages: + msg358723
2019-12-20 17:39:47brett.cannonsetstatus: open -> closed

nosy: + brett.cannon
messages: + msg358722

resolution: not a bug
stage: resolved
2019-12-20 16:42:56muralicreate