classification
Title: "in" should be consistent with return value of __contains__
Type: behavior Stage: resolved
Components: Documentation, Interpreter Core Versions: Python 3.7, Python 3.6, Python 3.5, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: docs@python Nosy List: Eric Lafontaine, Mariatta, berker.peksag, christian.heimes, docs@python, ezio.melotti, haypo, mark.dickinson, nparikh, r.david.murray
Priority: normal Keywords: easy

Created on 2012-09-24 00:13 by nparikh, last changed 2017-03-29 21:12 by Mariatta. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 152 merged aktech, 2017-02-18 09:53
PR 874 merged Mariatta, 2017-03-28 14:27
PR 875 merged Mariatta, 2017-03-28 14:27
PR 883 merged Mariatta, 2017-03-29 04:10
Messages (21)
msg171085 - (view) Author: Neal Parikh (nparikh) Date: 2012-09-24 00:13
The Python 2.7.3 documentation says the following about defining __contains__:

"Called to implement membership test operators. Should return true if item is in self, false otherwise. For mapping objects, this should consider the keys of the mapping rather than the values or the key-item pairs."

This suggests that while you should return True/False, you don't need to. Also, the documentation doesn't say that if you return a value other than True or False, it will silently coerce other values to be True or False when you use "in". For example:

>>> class Foo(object):
...     def __contains__(self, item): return 42
... 
>>> foo = Foo()
>>> 3 in foo
True
>>> foo.__contains__(3)
42

When __contains__ is defined, "in" should return whatever __contains__ returns, even if this value is neither True nor False. That would be consistent with, for example, the comparison operators: You can return anything from __lt__ and "x < 4" will correctly pass along that value regardless of what it is.
msg171086 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2012-09-24 00:26
The internal API is limited to flags as the sq_contains slot and the API function can only return an int:

  PyAPI_FUNC(int) PySequence_In(PyObject *o, PyObject *value)

In order to return the Python object we'd have to alter the API which we can't do as the type slot and the function is part of the stable API and ABI.

A doc update should explain why the code doesn't behave as you expect it.
msg171093 - (view) Author: Ezio Melotti (ezio.melotti) * (Python committer) Date: 2012-09-24 05:42
http://docs.python.org/py3k/reference/datamodel.html#object.__contains__ says that __contain__ "Should return true if item is in self, false otherwise.".  Here the lowercase true and false mean any true or false value, not just True and False and it's indeed possible to return any value.

The fact that 'in' only returns True/False, possibly converting the return value of __contains__, should be documented in the 'in' documentation.
http://docs.python.org/py3k/reference/expressions.html#membership-test-details says "For user-defined classes which define the __contains__() method, x in y is true if and only if y.__contains__(x) is true.".  This could be changed to say that the return value is converted to True/False.
msg171104 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2012-09-24 08:56
There was a related discussion on python-ideas a while back:

http://mail.python.org/pipermail/python-ideas/2010-July/007733.html
msg171167 - (view) Author: Neal Parikh (nparikh) Date: 2012-09-24 18:18
Thanks for passing along the thread, it was interesting. Oddly, it seemed to die off with no real resolution.

I realize it is now too late to change __contains__ to return a non-boolean value, but for reference, the reason I wanted to return something different from __contains__ was also to implement a DSL, though not a SQL-related one. Basically, I have some kind of abstract mathematical set S, and I wanted "x in S" to return a constraint that the variable x must lie in the set S for use in a larger problem description. (In fact, one could implement __contains__ so it returned True/False with a constant numeric argument and a constraint object with a variable argument.) This would have mirrored the way one would write all this mathematically.
msg171191 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2012-09-24 21:11
Do you need this PEP?
http://www.python.org/dev/peps/pep-0335/
msg171206 - (view) Author: Neal Parikh (nparikh) Date: 2012-09-24 23:53
I don't, but thanks for passing that along.
msg287265 - (view) Author: Eric Lafontaine (Eric Lafontaine) * Date: 2017-02-08 00:40
Hi,

For user-defined class, it's up to the class to do the right implementation in my opinion. It's true the description is wrong though. 

x in y means that x exist inside of y (so that the execution of y.__contain__(x) is executed successfully and (I guess) doesn't return None,False or 0).

I'll modify the doc to be :
For user-defined classes which define the __contains__() method, x in y is false if y.__contains__(x) is returning either None,False or 0.  Otherwise, x in y return true.

Regards,
Eric Lafontaine
msg287266 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-02-08 01:09
Eric: that is not precise enough, I'm afraid :)  See msg171093 for the correct documentation update.  Specifically, in returns True if __contains__ returns a true value, and False otherwise (not the difference in case, it matters).  There are more things than just None, False, and 0 that are false in Python.
msg287267 - (view) Author: Eric Lafontaine (Eric Lafontaine) * Date: 2017-02-08 01:25
Hi all,

Here are the test I've made to understand the behavior :

class Foo_42(object):
    def __contains__(self,item): return 42 
    
class Foo_neg(object):
    def __contains__(self,item): return -42 
    
class Foo_None(object):
    def __contains__(self,item): return  
    
class Foo_false(object):
    def __contains__(self,item): return False 
    
class Foo_true(object):
    def __contains__(self,item): return True

for foo in [Foo_false(),Foo_None(),Foo_neg(),Foo_true(),Foo_42()]:
    print("3 in foo:" + str(3 in foo))
    print("foo.__contains__(3)" + str(foo.__contains__(3)))
    

which output this :
3 in foo:False
foo.__contains__(3)False
3 in foo:False
foo.__contains__(3)None
3 in foo:True
foo.__contains__(3)-42
3 in foo:True
foo.__contains__(3)True
3 in foo:True
foo.__contains__(3)42

So as long as __contains__ return False or None, the 'in' operator will be False.  Otherwise true.
msg287268 - (view) Author: Eric Lafontaine (Eric Lafontaine) * Date: 2017-02-08 01:29
Hi David, sorry for the delay on my part for providing how I was getting to that conclusion.  I've also resurrected an old post as I want to start contributing more seriously :).

As this is documentation only (not changing the code behavior), I didn't take a look at the implementation of "in".

Could you enlighten me on what else would be considered "False" in this case?

Regards,
Eric Lafontaine
msg287269 - (view) Author: Eric Lafontaine (Eric Lafontaine) * Date: 2017-02-08 01:34
oh, I've got what you meant!

Proposed change :
For user-defined classes which define the __contains__() method, the in operator will convert to False "x in y" if y.__contains__(x) return False, 0 or None.  Otherwise, the in operator will return True for any other value being returned by y.__contains__(x).


Would that make more sense?
Regards,
Eric Lafontaine
msg287270 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-02-08 02:10
>>> bool(())
False
>>> bool([])
False
>>> bool('')
False

What you want to say is that 'in' coerces the result returned by __contains__ to a boolean value.
msg287271 - (view) Author: Eric Lafontaine (Eric Lafontaine) * Date: 2017-02-08 02:20
(first time trying to reply through email)

thanks for the example and you are right :
class Foo_emptylist(object):
    def __contains__(self,item): return []

class Foo_emptydict(object):
    def __contains__(self,item): return {}

class Foo_emptystring(object):
    def __contains__(self,item): return ''

for foo in
[Foo_false(),Foo_None(),Foo_emptylist(),Foo_emptydict(),Foo_emptystring(),Foo_neg(),Foo_true(),Foo_42()]:
    print("3 in foo:" + str(3 in foo))
    print("foo.__contains__(3)" + str(foo.__contains__(3)))

3 in foo:False
foo.__contains__(3)False
3 in foo:False
foo.__contains__(3)None
3 in foo:False
foo.__contains__(3)[]
3 in foo:False
foo.__contains__(3){}
3 in foo:False
foo.__contains__(3)
3 in foo:True
foo.__contains__(3)-42
3 in foo:True
foo.__contains__(3)True
3 in foo:True
foo.__contains__(3)42

So the proposition should be this then?
For user-defined classes which define a __contains__() method, the in
operator will apply bool() on the __contains__() method.  In other words,  "x
in y" is equivalent to "bool(y.__contains__(x))" and will return False if
bool(y.__contains__(x)) is equivalent to false.

Éric Lafontaine |  Membre du Projet VUE, Groupe Contrôle
Génie électrique, 54ème promotion UdeS | Étudiant en maitrise TI à l'ETS
VAS OPS chez Bell Mobility

« Nous voulons proposer une alternative de transport en présentant un
véhicule électrique spécifiquement conçu pour les déplacements urbains. »

2017-02-07 21:10 GMT-05:00 R. David Murray <report@bugs.python.org>:

>
> R. David Murray added the comment:
>
> >>> bool(())
> False
> >>> bool([])
> False
> >>> bool('')
> False
>
> What you want to say is that 'in' coerces the result returned by
> __contains__ to a boolean value.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue16011>
> _______________________________________
>
msg287306 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-02-08 12:08
You've got the right idea, but you are repeating yourself.  Keep it as short as possible while still conveying the correct information.  "coerce to boolean" is better than "apply bool", because the code may not in fact be using the bool function to do it.  Your "equivalent to" phrase would be OK as an alternative, but you only need to show the equivalence, no need to also explain it in words.
msg290720 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-28 14:13
New changeset 0ae7c8bd614d3aa1fcaf2d71a10ff1148c80d9b5 by R. David Murray (Amit Kumar) in branch 'master':
bpo-16011 clarify that 'in' always returns a boolean value
https://github.com/python/cpython/commit/0ae7c8bd614d3aa1fcaf2d71a10ff1148c80d9b5
msg290725 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-03-28 14:42
Thanks,
msg290739 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-03-28 16:32
New changeset 0957f262c5e47167efd520624557aebdc61bfda8 by Mariatta in branch '3.5':
bpo-16011: clarify that 'in' always returns a boolean value (GH-152) (GH-875)
https://github.com/python/cpython/commit/0957f262c5e47167efd520624557aebdc61bfda8
msg290740 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-03-28 16:33
New changeset c4021af50526f488c0c280e7c7eaa83ef80ae1df by Mariatta in branch '3.6':
bpo-16011: clarify that 'in' always returns a boolean value (GH-874)
https://github.com/python/cpython/commit/c4021af50526f488c0c280e7c7eaa83ef80ae1df
msg290791 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-03-29 21:10
New changeset fd704a02ca8713b0dae644f7f182f3e3d1218dbf by Mariatta in branch '2.7':
bpo-16011: clarify that 'in' always returns a boolean value (GH-152) (GH-883)
https://github.com/python/cpython/commit/fd704a02ca8713b0dae644f7f182f3e3d1218dbf
msg290792 - (view) Author: Mariatta Wijaya (Mariatta) * (Python committer) Date: 2017-03-29 21:12
Thanks everyone :) Patch has been merged and backported to 3.6, 3.5, and 2.7.
History
Date User Action Args
2017-03-29 21:12:55Mariattasetstatus: open -> closed
resolution: fixed
messages: + msg290792

stage: backport needed -> resolved
2017-03-29 21:10:16Mariattasetmessages: + msg290791
2017-03-29 04:10:48Mariattasetpull_requests: + pull_request785
2017-03-28 16:33:57Mariattasetmessages: + msg290740
2017-03-28 16:32:53Mariattasetnosy: + Mariatta
messages: + msg290739
2017-03-28 14:42:00r.david.murraysetmessages: + msg290725
stage: patch review -> backport needed
2017-03-28 14:27:54Mariattasetpull_requests: + pull_request773
2017-03-28 14:27:23Mariattasetpull_requests: + pull_request772
2017-03-28 14:13:03r.david.murraysetmessages: + msg290720
2017-02-22 02:06:01berker.peksagsetnosy: + berker.peksag
stage: needs patch -> patch review

versions: + Python 3.5, Python 3.6, Python 3.7, - Python 3.2, Python 3.3, Python 3.4
2017-02-18 09:53:29aktechsetpull_requests: + pull_request116
2017-02-08 12:08:18r.david.murraysetmessages: + msg287306
2017-02-08 02:20:48Eric Lafontainesetmessages: + msg287271
2017-02-08 02:10:39r.david.murraysetmessages: + msg287270
2017-02-08 01:34:45Eric Lafontainesetmessages: + msg287269
2017-02-08 01:29:06Eric Lafontainesetmessages: + msg287268
2017-02-08 01:25:04Eric Lafontainesetmessages: + msg287267
2017-02-08 01:09:23r.david.murraysetnosy: + r.david.murray
messages: + msg287266
2017-02-08 00:40:41Eric Lafontainesetnosy: + Eric Lafontaine
messages: + msg287265
2012-09-24 23:53:59nparikhsetmessages: + msg171206
2012-09-24 21:11:17hayposetnosy: + haypo
messages: + msg171191
2012-09-24 18:18:19nparikhsetmessages: + msg171167
2012-09-24 08:56:24mark.dickinsonsetnosy: + mark.dickinson
messages: + msg171104
2012-09-24 05:42:20ezio.melottisetversions: + Python 3.2, Python 3.3, Python 3.4
nosy: + ezio.melotti

messages: + msg171093

keywords: + easy
stage: needs patch
2012-09-24 00:26:10christian.heimessetnosy: + christian.heimes
messages: + msg171086
2012-09-24 00:13:51nparikhcreate