Current 3.2 doc, 5.9. Comparisons, has this paragraph about mixed-type comparisons.

"The operators <, >, ==, >=, <=, and != compare the values of two objects. The objects need not have the same type. If both are numbers, they are converted to a common type. Otherwise, the == and != operators *always* consider objects of different types to be unequal, while the <, >, >= and <= operators raise a TypeError when comparing objects of different types that do not implement these operators for the given pair of types. You can control comparison behavior of objects of non-built-in types by defining rich comparison methods like __gt__(), described in section Basic customization."

Sentence 3: "If both are numbers, they are converted to a common type." I suspect it would be more true to say 'common internal type' as I would not think it a language requirement to produce Python objects.

In any case, I think it is only true for built-in number types, and I do not see that qualification anywhere previously.

That aside, it does not appear to be true for Decimals and Fractions in 2.7.1.

Sentence 4: first clause is only true for built-in types. That qualification is not obvious to everyone, as evidenced by a current python-list sub thread.

For 2.7, which has a different continuation, I suggest adding 'built-in' before 'objects of'.
For 3.2/3, I suggest deleting '*always*' and adding a comma after 'TypeError' so that the 'when' condition applies to equality comparisons also.

After discussion about same-type comparisons, there is another paragraph about mixed-type comparison:

"Comparison of objects of the differing types depends on whether either of the types provide explicit support for the comparison. Most numeric types can be compared with one another, but comparisons of float and Decimal are not supported to avoid the inevitable confusion arising from representation issues such as float('1.1') being inexactly represented and therefore not exactly equal to Decimal('1.1') which is. When cross-type comparison is not supported, the comparison method returns NotImplemented. This can create the illusion of non-transitivity between supported cross-type comparisons and unsupported comparisons. For example, Decimal(2) == 2 and 2 == float(2) but Decimal(2) != float(2)."

I suggest deleting this entirely. The first sentence and first clause of the second repeat what was said above. The rest is obsolete as float/decimal comparisons *are* implemented in 2.7.1 and 3.2.0.

Can you provide a patch?

[Docs]
"If both are numbers, they are converted to a common type."

[Terry]
"In any case, I think it is only true for built-in number types,"

It's not even true for built-in number types.  When comparing an int with a float, it's definitely *not* the case that the int is converted to a float and the floats compared.  And that's for good reason:  the int -> float conversion is lossy for large integers, so if int <-> float comparisons just converted the int to a float before comparing, we'd have (for example):

>>> 10**16 == 1e16 == 10**16 + 1

leading to broken transitivity of equality, and strange dict and set behaviour.

So int <-> float comparisons do a complicated dance under the hood to compare the exact numerical values of the two objects and produce the correct result.

I'm not sure what the intent of the original sentence was, or how to reword it.  The key point is simply that it *is* possible to compare an int with a float, and that the result is sensible, based on numeric values.

Would it be ok to state that:
1) <, >, ==, >=, <=, and != compare the values of two objects;
2) the two objects don't necessarily have to be of the same type;
3) with == and !=, objects of different types compare unequal, unless they define a specific __eq__ and/or __ne__;
4) with <, >, <=, and >=, the comparison of objects of different types raises a TypeError, unless they define specific __lt__, __gt__, __le__, and __ge__;
5) some built-in types define these operations, so it's possible to compare e.g. int and floats;

This should summarize the possible behaviors.  There's no reason IMHO to expose implementation details and to special case built-in types (unless their comparison is actually different and doesn't depend on __eq__, __ne__, etc.).

In Python 3, where all classes inherit from object, the default rules are, by experiment, (which someone can verify from the code) simpler than you stated.

3. By default, == and /= compare identities.
4. By default, order comparisons raise TypeError.
ob <= ob raises even though ob == ob because ob is ob.

I am not sure of the method look-up rules for rich comparisons, but perhaps the following are true:

3) with == and !=, an object is equal to itself and different objects (a is not b) compare unequal, unless the class of the first define a specific __eq__ and __ne__;

4) with <, >, <=, and >=, comparison raises a TypeError, unless the class of the first object defines specific __lt__, __gt__, __le__, and __ge__, or the class of the second defines the reflected method (__ge__ reflects __lt__, etcetera);

What is not clear to me is whether the reflected method is called if the first raises TypeError. The special method names doc (reference 3.3) says "A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments. 
...
There are no swapped-argument versions of these methods (to be used when the left argument does not support the operation but the right argument does); rather, __lt__() and __gt__() are each other’s reflection, __le__() and __ge__() are each other’s reflection, and __eq__() and __ne__() are their own reflection."

Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section.

I think point 5 should say a bit more: builtin numbers compare as expected, even when of different types; builtin sequences compare lexicographically.

http://bugs.python.org/issue15997 is this issue related to what Terry has mentioned:

Does 'not supported' mean 'raises TypeError', 'returns NotImplemented', or both? If the last, I don't really understand the reason for NotImplemented versus TypeError. That point should be clarified in 3.3 also. And 3.3 should be referenced in the comparisons section.

I am working on this patch and need confirmation as to whether or not this has to be included in my patch. I'm not clear on it so I may just pass on making a patch if it is required for this issue.

After further thought: This section is about the syntax operators, not the special methods. The syntax operators never evaluate to NotImplemented as they (apparently) interpret its return from a special method the same as a raising of TypeError, and always raise TypeError when neither the op or its reflection is supported. So there should be no mention of NotImplemented here. Just a reference to 3.3. #15997 is related to my 'wonder' but not directly relevant to a patch for this. Please submit a draft patch when you have one.

I determined that 'raise TypeError' and 'return NotImplemented' both result in the call of the reflected method, at least for a couple of cases. (And same seems true for arithmetic ops too.)

class C():
    def __ge__(self, other):
        # print("in C.__ge__", end='')
        return True
    def __add__(self, other):
        return 44
    __radd__ = __add__

class O():
    def __le__(self, other):
        # print ("in O.__le__")
        return NotImplemented
    def __add__(self, other):
        return NotImplemented
    
c = C()
o = O()
ob = object() 
print(c >= o, o <= c, ob <= c)
# True True True
# print(ob <= ob) # raises TypeError
print(c + o, o + c, ob + c)
# 44 44 44
# print(ob + ob)  # raises TypeError
# print(ob >= o)  # with O.__le__ print uncommented
# in O.__le__  # so non-implemented reflected o <= ob *is* called
# TypeError: unorderable types: object() >= O()

I've attempted to incorporate both Terry's and Ezio's suggestions. Here is a patch to get started with. There is a section that has been deleted. Patch uploaded.

Some minor comments:

-The operators ``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the
+``<``, ``>``, ``==``, ``>=``, ``<=``, and ``!=`` compare the values of two 

I think it reads better to start a sentence (and in this case a paragraph) with a word rather than a symbol.

-values of two objects.  The objects need not have the same type. If both are
+objects. The two objects don't necessarily have to be of the same type. With 

The replacement sentence seems wordier to me.

+(:meth:`__ge__()` reflects :meth:`__lt__()`, etcetera). Builtin numbers compare 
+as expected, even when of different types. Builtin sequences compare 

"Built-in" is hyphenated in the docs.  See, for example, here:

http://docs.python.org/dev/library/functions.html

> I determined that 'raise TypeError' and 'return NotImplemented' both
> result in the call of the reflected method

Are you sure?  raise TypeError *should* result in the operation being abandoned, with the reflected operation not tried.


Python 3.3.0rc2+ (default:3504cbb3e1d8, Sep 20 2012, 22:08:44) 
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> class A:
...     def __add__(self, other):
...         raise TypeError("Don't know how to add")
...     def __le__(self, other):
...         raise TypeError("Can't compare")
... 
[65945 refs]
>>> class B:
...     def __radd__(self, other):
...         return 42
...     def __ge__(self, other):
...         return False
... 
[66016 refs]
>>> A() <= B()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 5, in __le__
TypeError: Can't compare
[66064 refs]
>>> A() + B()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 3, in __add__
TypeError: Don't know how to add
[66065 refs]

You are right, I did not look deep enough. I was fooled by the conversion of NotImplemented, returned from object.__le__, etc, to TypeError. Sorry for that noise.

For comparison and arithmetic, the actual alternative to defining a function that returns NotImplemented seems to be to not define it at all.

class C():
    def __ge__(self, other):
        return True
    def __add__(self, other):
        return 44
    __radd__ = __add__

class O():
    pass  # removed NotImplemented defs
    
c = C()
o = O()
print(c >= o, o <= c)
# True True
print(c + o, o + c)
# 44 44

(I looked at the codes for binary_op1 in abstract.c and do_richcompare in object.c and do not yet see any effective difference between not defined and a NotImplemented return.)

I'll take a look at the patch later.

Changed patch to include suggestions by Chris Jerdonek.

http://bugs.python.org/issue12067#msg170953

Considering that the docs have changed does this issue still need to be open?

Hi, I'd like to revive this issue.

IMHO, the changes in issue12067-expressions_v2.diff go too far. I don't think that deleting the entire section about the details of comparing objects of the same type makes sense.

I agree with Terry's statement in msg170936 that the chapter is about the operators and not about the ways to customize them, so some of what the patch introduces in that area should not be introduced.

So far, that means that I'm pretty much against that patch entirely...

Having said that, I do believe that there are still issues:

1. both the 2.7 and 3.x sections about the comparison operators are sufficiently convoluted and could be organized better by grouping the various statements that are made, into categories like this:

- comparisons involving objects of user-defined types
- comparing objects of same built-in type
- comparing objects of differing built-in type

2. There are still some errors, ambiguities and omissions that need to be fixed. For example, in the 3.x version:

a) omission about treatment of NaN for numbers of different type (could in theory be implied from the statement "are converted to same type", but that statement is problematic as was pointed out in Mark's comment on this issue).

b) Amgiguous statement "Most numeric types [of same type] can be compared with one another.". I think what is true is that all built-in numeric types (including Fraction and Decimal) compare mathematically correct in 3.x, and that the only non-support is that complex numbers are not considered orderable and an attempt to use an ordering operator raises TypeError.

c) Ambiguous statement "When cross-type comparison is not supported, the comparison method returns NotImplemented.". If this is about the customization methods, it should not be here, but there. Here, it is relevant that a TypeError is raised when using the operator.

d) Terminology in "Bytes objects are compared lexicographically using the numeric values of their elements.": Chapter [4.8.1. Bytes] defines bytes objects as immutable sequences of single bytes (not elements).

e) Terminology in "Tuples and lists are compared lexicographically using comparison of corresponding elements.": lists and tuples contain "items" not "elements", and an item-wise comparison should not be called "lexicographically" because that makes sense only when the items are characters.

f) Ambiguity in "If not equal, the sequences are ordered the same as their first differing elements.": "Are ordered" could be interpreted (e.g. by non-native speakers) to mean that the sequence is changed to achieve that ordering, which is not the case of course.

g) Editorial: In the list item about sets and froze sets, the example set {2,3} is not in example font.

h) Omission: Range types are not covered.

i) Omission: The section "Comparison of objects of differing types..." is silent about which built-in types support comparison across types (except for numeric types where that is covered). I think that should be explicitly listed for those built-in types that are also listed explicitly in the section about comparing objects of same type.

I'll try to come up with a patch for 3.x, and once that is agreed, with one for 2.x.

Andy

Uploaded issue12067-expressions_v3.diff for the 3.5 tip.
Please review.

Uploaded issue12067-expressions_v4.diff to improve the unicode footnote 3, and to revert to using the term "lexicographical" for sequences (after learning that it applies there as well). Also, this version was produced using "hg diff" to make it properly reviewable.

Please review.

PS: The v4 patch does not address comments f) and h) from msg222204, and it seems to me they do not need to be addressed.

Terry,
I'd like to comment on your statement:
> 3. By default, == and /= compare identities.
in msg148774.

What experiment lead you to that conclusion?

Here is one that contradicts it (using cpython 3.4.1):

>>> i1 = 42
>>> f1 = 42.0
>>> i1 == f1
True
>>> i1 is f1
False

Is it possible, that your experiment got influenced by the optimization that attempts to reuse existing objects of immutable types?
Like in this:

>>> i1 = 42
>>> i2 = 40 + 2
>>> i1 == i2
True
>>> i1 is i2
True

Andy

Uploaded v5 of the patch.

Changes:

1. The statement that comparison of different built-in types (always) raises TypeError, was too general. Changed to distinguish equal and order operators, as summarized by Ezio in items 3) and 4) of msg148760.

2. Ensured max line length of 80, in text areas affected by the patch.

Andy

It seems I still need to practice creating patches ... uploading v6 which should create a review link. No other changes.
Sorry for that.
Andy

Another attempt. Really sorry...

In py3, *everything* is an instance of class object. This makes like simple than in 2.x. The default comparison rules are set by the rich comparison methods of object. 'By experiment' meant by experiments with instances of object, which use those default methods, rather than by inspection of the relevant .c source code. Instances of subclasses taht do not override the defaults would act the same. Here is what seem to be the default code, from object.c, do_compare.  It verifies what I said (v, w are pointers, which represent identity):

    /* If neither object implements it, provide a sensible default
       for == and !=, but raise an exception for ordering. */
    switch (op) {
    case Py_EQ:
        res = (v == w) ? Py_True : Py_False;
        break;
    case Py_NE:
        res = (v != w) ? Py_True : Py_False;
        break;
    default:
        /* XXX Special-case None so it doesn't show as NoneType() */
        PyErr_Format(PyExc_TypeError,
                     "unorderable types: %.100s() %s %.100s()",
                     v->ob_type->tp_name,
                     opstrings[op],
                     w->ob_type->tp_name);
        return NULL;
    }
    Py_INCREF(res);
    return res;

Subclasses can and ofter do override the default methods. In particular, the number subclasses compare by value, across number types.

I see.
But I don't think it is a sensible default, as the source code states.

The Python doc (v2 and v3) is quite consistent in stating that `==` compares the values of two objects, while `is` compares object identity.

Having a default implementation on the object type that implements `==` by comparing object identity is not consistent with that.

-> Can someone please elaborate what the reason for that is?

-> Where is the discrepancy between the documentation of == and its default implementation on object documented?

To me, a sensible default implementation for == on object would be (in Python):

  if v is w:
    return True;
  elif type(v) != type(w):
    return False
  else:
    raise ValueError("Equality cannot be determined in default implementation")

Andy

Uploaded v8 of the patch for 3.4 and default.

It reflects hopefully everything that was said in this issue thread, and on the python-dev mailing list (subject: == on object tests identity in 3.x), at least to the extent it was related to comparisons.

Besides the doc changes it contained previously, it now also contains improvements for the test suite for comparisons (lib/test/test_compare.py).

-> Please review both.
 
Andy

+  In other words, the following expressions should have the same result:
+
+    ``x == y`` and ``not x != y``
+
+    ``x < y`` and ``not x >= y``
+
+    ``x > y`` and ``not x <= y``

I think the second and third items here go too far: sets don't obey these rules, for example.  Not all uses of comparisons need to force a total ordering.

OTOH, you leave out a more fundamental relation, namely that `x < y` and `y > x` should ordinarily give the same result, as should `x <= y` and `y >= x`.

Mark: Both are good points!

Would you add the cases from your second comment under "symmetry"?

Uploaded v9 of the patch for 3.4 and default.

It reflects Marc's comment, plus the result of the recent discussion on python-dev since v8 of th epatch, up to 2014-07-15 (subject: == on object tests identity in 3.x).

-> Please review the patch.

- This bug should discuss doc updates, not question the rules.

- The rules have evolved over time and the docs stayed behind.

- We should definitely update the 2.7 docs as well as the 3.4 and 3.5 (in development) docs.  The 2.7 docs need to be different than the 3.x docs.

- The language reference manual should clearly state the rules so that implementers can use them as guidelines for implementation.

- There are several sets of relevant rules:

(a) How is each operator translated into a series of lookups and method calls, etc.  It's similar to other binary operators except that the reverse for __lt__ is __gt__ instead of __rlt__, and there's an extra rule that if __ne__ doesn't exist we compute __eq__ and take the opposite.

(b) The default implementation (e.g. default == falls back to 'is', < raises TypeError).

(c) The rules for built-in types, especially numbers (if there are still special cases that aren't explained by the __xx__ methods on the various numeric types).

The point about “a != b” deferring to “not a.__eq__(b)” is not documented anywhere that I am aware of. In fact the opposite is currently documented at <https://docs.python.org/release/3.4.0/reference/datamodel.html#richcmpfuncs>, so maybe this needs to be fixed, one way or another.

That's a pretty new feature. Someone probably forgot to clean up all the
places where it was documented.

On Sat, Sep 6, 2014 at 4:18 PM, Martin Panter <report@bugs.python.org>
wrote:

>
> Martin Panter added the comment:
>
> The point about “a != b” deferring to “not a.__eq__(b)” is not documented
> anywhere that I am aware of. In fact the opposite is currently documented
> at <
> https://docs.python.org/release/3.4.0/reference/datamodel.html#richcmpfuncs>,
> so maybe this needs to be fixed, one way or another.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue12067>
> _______________________________________
>

Just wanted to say that i will continue working on this, working in the comments made so far...
Andy

Maybe it would be wise to split this task up and commit the bits that don’t need any more work. I think the existing patch might already solve Issue 22001.

@Guido:
Agree to all you said in your #msg226496.

There is additional information about comparison in:
- Tutorial (5.8. Comparing Sequences and Other Types),
- Library Reference (5.3. Comparisons),
- Language Reference (3.3.1. Basic customization)
that needs to be reviewed in light of this patch.

I'm just not sure I want to make this patch even larger as it is already, and tend to do that in a follow on issue and patch (unless directed otherwise).

Andy

Uploading v10 of the patch, which addresses all review comments made on v9.

There is one open question back to Martin Panter about which different types of byte sequences can be compared in Py 3.4.

I also believe this patch addresses all of Issue 22001. Let me know if you find that that is not the case.

If we continue to scope this patch to only the comparison chapter of the language reference, then I think we are done (see msg229229 about other places that need review and possibly updates).

Please review the patch v10.

Here is the delta between v9 and v10 of the patch, if people want to see just that.

About the byte sequence comparisons, I wondered if it was misleading to say that a list(), tuple() or range() can only be compared to the same type, without mentioning that bytes() and bytearray() can be compared to each other.

BTW just noticed you say range() supports lexicographical ordering, which I think is incorrect.

I have addressed the comments by Jim Jewett, Martin Panter and of myself in a new version v11, which got posted.

For the expression.rst doc file, this version of the patch has its diff sections in a logical order, so that the original text and the patched text are close by each other.

Please review.

I also made sure in both files that the line length of any changed or new lines is max 80. Sorry if that creates extra changes when looking at deltas between change sets.

I have posted v12 of the patch, which addresses all comments since v11.

This Python 3.4 patch can be applied to the "default" (3.5 dev) branch as well.

I will start working on a similar patch for Python 2.7 now.

Issue 4395 is already open for the != delegating to __eq__ issue that Guido pointed out earlier.

Yet another issue that this doc patch should solve: Issue 22000.

I am posting v13 of the patch that works with the current “default” (3.5) branch. Minor modifications to the documentation part:

* Undo some paragraph reflowing to reduce the size of the diff hunks
* Use math.nan instead of nan = float('NaN')
* Tweaked the new subheadings to better differentiate “Value comparisons” (which this bug is concerned with) from “Identity comparisons”

Changes I made to the added test cases:

* Drop Python 3 version check
* Use TestCase.subTest() so it’s easier to see everything that [is] was failing
* Merge some tests for != and fix changed expectations due to Issue 21408 being fixed
* Drop _inst_str() and __str__(), apparently unused

It would be nice to see at least the documentation part reviewed and committed, even if the tests require more work. (After having to hack the tests to get them working, I might point out some odd bits in the code review.) Though I guess they are just tests, so it doesn’t matter so much as long as they pass.

I have posted v14 of the patch (for the 3.5 'default' branch), based on Martin's v13. v14 addresses all comments Martin made, as described in my responses to them (see patch set 10).

On Issue 4395: That issue should be pursued in addition to this issue; it seems Martin's patch for it is complementary to the patch for this issue here.

On Issue 22000: I agree that that issue is addressed by the patch for this issue here.

All: Please review the v14 patch.

Patch v15. No doc changes, but I refactored the test code:

* Manually merged with recent changes
* Separate assert_equality_only() and assert_total_order() test methods. Hopefully this is a bit simpler for people to understand and review, and avoids suggesting that partial ordering is tested.
* Dropped subTest() instances with identical parameters. The basic stack trace can already distinguish these.
* Eliminated is_value_comparable(); remove the ”meth” parameters from the call sites instead

I think we can commit documentation and tests separately. I just did a quick review of the test changes and I will add some review comments later (sorry, lack of time :)).

I can split out a documentation-only patch if it would help get that committed.

In the meantime, patch v16 includes some fixups to comments etc in the test code that I missed myself.

New changeset 1fc049e5ec14 by Martin Panter in branch '3.4':
Issue #12067: Rewrite Comparisons section in the language reference
https://hg.python.org/cpython/rev/1fc049e5ec14

New changeset b6698c00265b by Martin Panter in branch '3.5':
Issue #12067: Merge comparisons doc from 3.4 into 3.5
https://hg.python.org/cpython/rev/b6698c00265b

New changeset 294b8a7957e9 by Martin Panter in branch 'default':
Issue #12067: Merge comparisons doc from 3.5
https://hg.python.org/cpython/rev/294b8a7957e9

I committed the changes to expressions.rst for 3.4+. That still leaves the changes to test_compare.py, and possibly changes for 2.7.

Andy: In msg229721 you mentioned a potential 2.7 patch. Did you get anywhere with that? Even if it is only half finished, someone else may be able to keep working on it.

Here is a port of the documentation to Python 2. Main differences:

* Default rules for order comparisons are different
* Not all kinds of objects inherit from object()
* str(), unicode() compatibility
* xrange() only seems to have default comparability
* NAN, “binary sequences” and sets not listed

Updated patch for 2.7, which I plan to commit soon. Corresponding Py 3 patch coming soon.

New changeset 8c9a86aa222e by Martin Panter in branch '3.5':
Issue #12067: Recommend that hash and equality be consistent
https://hg.python.org/cpython/rev/8c9a86aa222e

New changeset 9702c5f08df1 by Martin Panter in branch '3.6':
Issues #12067: Merge hash recommendation from 3.5
https://hg.python.org/cpython/rev/9702c5f08df1

New changeset 9dbb7bbc1449 by Martin Panter in branch 'default':
Issues #12067: Merge hash recommendation from 3.6
https://hg.python.org/cpython/rev/9dbb7bbc1449

New changeset 8a9904c5cb1d by Martin Panter in branch '2.7':
Issue #12067: Rewrite Comparisons section in the language reference
https://hg.python.org/cpython/rev/8a9904c5cb1d

It appears all the patches for this issue have been applied.  Is the only open item the changes to test_compare?

Yes I think I committed all the documentation. Someone needs to decide whether to use Andy’s tests as they are, or perhaps modify or drop some or all of them.

I've created a PR for the changes to test_compare from v16 of the patch.

Martin: [tests need review].  Yep.  Improved constant folding and elimination of duplication now causes many 'is not' tests to fail.  Language tests should not test current implementation limitations.