classification
Title: Document the effects of NotImplemented on == and !=
Type: behavior Stage:
Components: Documentation Versions: Python 3.0, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: facundobatista, georg.brandl, mark.dickinson, rhettinger, terry.reedy
Priority: low Keywords:

Created on 2008-10-09 13:39 by mark.dickinson, last changed 2008-11-21 09:30 by mark.dickinson. This issue is now closed.

Messages (10)
msg74576 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-10-09 13:39
The Decimal module breaks transitivity of equality:  Decimal(2) == 2 and 
2 == float(2), but Decimal(2) != float(2).

The Python set and dict implementations rely on transitivity of equality 
for correct operation.

These two facts together give some strange results when playing with 
sets and dicts involving Decimals and floats.  For example (with Python 
2.6):

>>> s = set([Decimal(2), float(2)])
>>> t = set([2])
>>> s | t == t | s
False
>>> len(s | t)
2
>>> len(t | s)
1

Other strange examples, and possible solutions, were discussed recently 
on comp.lang.python;  see the thread starting at:

http://mail.python.org/pipermail/python-list/2008-September/508859.html

Possible solutions:

(1) Document the problem, making it clear in the language reference that 
correct set operation relies on transitivity of equality, and adding a 
note to the decimal documentation indicating that mixing floats and 
Decimals in a container is asking for trouble.

(2) Fix up Decimal so that equal numeric objects compare equal; this 
would also involve fixing the hash operation.  To me, this goes against 
the philosophy of keeping the Decimal module close to the specification.

(3) Terry Reedy suggested (in the c.l.python thread linked to above) a 
simpler version of (2): allow Decimal(i) == float(i) or Decimal(i) == 
Fraction(i) to return True for all integers i.  (Decimal('0.5') == 0.5 
would still return False.)  This fixes transitivity, and has the 
advantage of not requiring any changes to the hash functions:  
hash(Decimal(i)) == hash(float(i)) is already true for all integers i.

My own preference would be simply to document the problem;  it doesn't 
seem like something that's going to affect that vast majority of Python 
users.

Raymond, Facundo:  any thoughts?
msg74586 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-10-09 16:45
There are two issues involved:
1. documenting set behavior
2. what to do, if anything, about Decimals and other numbers

Since users are free to create similar problems, and since sets are
missing from the Reference section on comparisons, I opened a separate
set documentation issue http://bugs.python.org/issue4090
leaving this as a Decimal-other_number equality issue.

The root of the problem is that all members of s are members of t and
vice versa.  This should make s and t equal, but they are not.  This
also breaks the definitions of issubset (<=), issuperset (>=), union
(|), and symmetric_difference (^) as shown in the c.l.p thread.

Transitivity is also fundamental in logic and the rule of substitution.
 So I strongly prefer that it be preserved in Python as released.

Another way to restore transitivity is
(4) make Decimal(1) != int(1) just as Decimal(1) != float(1).  Having a
Decimal be equal in value to just one of two things that are equal in
value is incoherent, and leads to incoherent results.
msg74587 - (view) Author: Facundo Batista (facundobatista) * (Python committer) Date: 2008-10-09 17:05
(Ok, remember that I'm not a "numeric" guy before start hitting me, :p )

I think that if we have Decimal(1)==1, and 1==1.0, to have Decimal(1)==1.0.

We always rejected comparison with "unsupported types", but having this
situation, I'd propose to put something like the following at the
beggining of __eq__() and similars:

    def __eq__(self, other):
        if isinstance(other, float) and int(other)==other:
            other = int(other)

What do you think?
msg74588 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-10-09 17:13
Recommend not "doing anything" about decimals and other numbers.  What
you're seeing is a predictable consequence of NotImplemented being
returned by some but not all cross type comparisons.  IMO, it is
perfectly reasonable that both decimals and floats can be compared to
integers but not to each other.  Integers are a "universal donor" in
this respect but the two float types are not.

It is true that equality should be transitive but the same cannot be
said for the *ability of types* to be compared.  Unfortunately, the ==
operator masks what is going on by returning False instead of raising a
NotImplementedError.  IOW, it the apparant loss of transitivity when
"float(2) == Decimal(2)" returns False is an illusion; instead, the
False return means that the types cannot be compared at all.

If any doc changes are made with respect to this issue, it should be in
the docs for the == and != operators and for NotImplemented.
msg74590 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-10-09 18:06
If Decimal(2) == float(2) were to raise an error,
set([Decimal(2), float(2)]) would fail, as I would argue it ought to,
and the set anomalies would disappear.
msg74612 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-10-10 00:20
I'll take up the doc fix for the current state of affairs.

A change from returning NotImplemented to raising NotImplementedError
would need be a separate feature request or PEP.

Alternatively, we could decide to allow decimal/float comparisons -- the
float can be converted to a decimal exactly and compared exactly -- it
would be slow but it would work and have precise semantics.

Am resetting the priority back to low because the current behavior is
exactly what is supposed to happen.  It is an automatic consequence of
how we use NotImplemented and the decision to allow integer/float and
integer/decimal comparisons but not float/decimal comparisons.  There is
not bug here, just something that offends ones sensibilities.
msg74631 - (view) Author: Facundo Batista (facundobatista) * (Python committer) Date: 2008-10-10 12:21
2008/10/9 Raymond Hettinger <report@bugs.python.org>:

> Alternatively, we could decide to allow decimal/float 
> comparisons -- the float can be converted to a decimal
> exactly and compared exactly -- it would be slow but 
> it would work and have precise semantics.

-0

Note that this could lead to surprising behaviours, when doing these
comparations... Decimal("1.1")==1.1 will be true, buy maybe
Decimal("1.235445687")==1.235445687 will not (I didn't try if this
particular comparison will fail, but hope you get the idea).

This is why I suggested the other way... we now allow comparison to
integers, let's allow comparisons when the floats are equal to the
integers, and no more.
msg74641 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2008-10-10 19:57
More sensibility offenders:

>>> s = {fractions.Fraction(17,1), decimal.Decimal(17)}
>>> s-{17}
set()

Removing one thing removes two.

>>> s.remove(17)
>>> 17 in s
True

Removing something leaves it there.

>>> s
{Fraction(17, 1)} # random choice

Removing one thing and subtracting the set with that one thing give
different results.

>>> s = {decimal.Decimal(17), fractions.Fraction(17,1)}
>>> s.remove(17)
>>> s
{Decimal('17')}

The behavior of 'set' s depends on the order items are added.

> Facundo's suggested code:

    if isinstance(other, float) and int(other)==other:
        other = int(other)

would be more efficient, I assume as

    if isinstance(other,float):
        ifloat = int(other)
        if other == ifloat:
            other = ifloat

or if the CAPI has an efficient 'float_isint' function that accesses the
bits of a float, as the C equivalent of 

    if isinstance(other, float) and float_isint(other):
        other = int(other)

I remember float-Decimal comparison being rejected/deferred in part for
being problematical for fractional values.  That is why I suggested
implementing it, at least for the present, for integral floats (and
Fractions) only, which are relatively easy to detect.
msg75990 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2008-11-17 22:56
r67249
msg76168 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-11-21 09:30
Looks good.  Thanks, Raymond.

Minor typos:  "numberic" -> "numeric"
"the expression ``x in y`` equivalent to" -> 
"the expression ``x in y`` is equivalent to"
History
Date User Action Args
2008-11-21 09:30:32mark.dickinsonsetmessages: + msg76168
2008-11-17 22:56:43rhettingersetstatus: open -> closed
resolution: fixed
messages: + msg75990
2008-10-10 19:57:33terry.reedysetmessages: + msg74641
2008-10-10 12:21:20facundobatistasetmessages: + msg74631
2008-10-10 00:20:07rhettingersetpriority: normal -> low
assignee: rhettinger
messages: + msg74612
2008-10-09 18:06:28terry.reedysetpriority: low -> normal
assignee: georg.brandl -> (no value)
messages: + msg74590
2008-10-09 17:13:46rhettingersettitle: equality involving Decimals is not transitive; strange set behaviour results -> Document the effects of NotImplemented on == and !=
nosy: + georg.brandl
messages: + msg74588
priority: normal -> low
assignee: georg.brandl
components: + Documentation
2008-10-09 17:05:05facundobatistasetmessages: + msg74587
2008-10-09 16:45:09terry.reedysetmessages: + msg74586
2008-10-09 13:39:43mark.dickinsoncreate