classification
Title: Difference in behavior between set() and collections.abc.MutableSet() derived objects
Type: Stage: resolved
Components: Library (Lib) Versions: Python 3.6
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: rhettinger Nosy List: Horacio Hoyos, ned.deily, r.david.murray, rhettinger
Priority: normal Keywords:

Created on 2017-04-23 18:07 by Horacio Hoyos, last changed 2017-04-28 03:54 by rhettinger. This issue is now closed.

Files
File name Uploaded Description Edit
example.py Horacio Hoyos, 2017-04-23 19:47 Example script
Messages (10)
msg292175 - (view) Author: Horacio Hoyos (Horacio Hoyos) Date: 2017-04-23 18:07
Hi, attempt 2.

My system is MacOs Yosemite (10.10.5), I have installed Python 3.6.1 downloaded from the official Python website.

I was having issues while testing a custom Set implementation using the _collections_abc base MutableSet and found that my issue was apparently resolved with issue 8743. From the fix, in the attached script I would expect the and operation between my set implementation and a string to fail with a TypeError, given that string is not an instance of Set. However, the error is not raised, i.e. the print statement is executed. 

From the discussion on issue 8743, I would expect _collections_abc.py to have a test for Set instances, but is not the case (for example):

    def __and__(self, other):
        if not isinstance(other, Iterable):
            return NotImplemented
        return self._from_iterable(value for value in other if value in self)

That is, I was expecting a isinstance(other, Set) somewhere there.

In my previous post I was told my python installation was broken. However, I checked the collections_abc.py in my windows system and is the same. I am not an expert on the Python build system, but the patch in bug 8743 applies to /Lib/_abcoll.py, but I guess _collections_abc.py is generated somehow.
msg292178 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-04-23 19:00
You don't appear to have successfully attached the script.  Can you try again please?

The patching in the issue you point to does not patch __and__, it just sets it equal to __rand__.
msg292180 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-04-23 19:25
And rather than focusing on what lines of code you believe are missing, please provide a complete script or lines of Python code with any instructions on how to run them, the actual output you see when you run it, and what you expect the output should be.  Otherwise, we just have to guess at what the issue here is and we'll probably guess wrong.
msg292182 - (view) Author: Horacio Hoyos (Horacio Hoyos) Date: 2017-04-23 19:47
Guess my second attempt didn't work as expected ;-).

I have attached the file now. To run just invoke the script:

$ python3 example.py
s&t did not screen-out general iterables

I would expect to see no output.
msg292186 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2017-04-23 21:14
OK, thanks for supplying a concrete example.  From it, it's clear that it has nothing specific to do with Python on macOS.  Adding Raymond as maintainer of sets.
msg292312 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-04-26 07:22
> I would expect the and operation between my set implementation 
> and a string to fail with a TypeError, given that string is 
> not an instance of Set. However, the error is not raised ...

Odd as it may seem, this isn't a bug; rather it was a deliberate design decision by GvR.  Regular sets have a intersection() method that accepts any iterable.  In contrast, the Set abstract base class doesn't have a intersection() method and instead uses __and__() to do the work.

While there is room to have debated whether that design decision was a good one, that ship sailed long ago.  Tightening the input requirement now would almost certainly break existing code and would take away needed functionality.

Alternatively, we could liberalize set.__and__ to make it accept any iterable (to match the ABC), but Guido advised against this based on his bad experiences with list.__iadd__ accepting any iterable (for example:   s = ['abc', 'def']; s += 'def').  I think he's right that such an amendment would be a bug factory.

You'll likely find a number of places where the collection ABCs differ in minor details from various concrete classes.  In almost every case, you'll find a reason for the choices that were made.  It hasn't been a problem in practice unless someone (like the OP) expects that the ABCs have higher fidelity and can used to reimplement all of the details of the related concrete class (i.e. expecting to use Set() to match all the concrete implement nuances tested in Lib/test_set.py).
msg292467 - (view) Author: Horacio Hoyos (Horacio Hoyos) Date: 2017-04-27 17:59
Although I agree that some design decisions forge a language and sometimes these historical developments win over sounder or more "logical" arguments.

That being said, I believe (as the OP of issue 8743) that the ABC collections should comply to the API of the Python base collections. In that sense, I would expect any custom collection implemented by extending the ABC collections to be used in place of an exiting base one, without any "hiccups". Thus, the custom collection should pass all the tests in the Lib/tests.

Of course, for the moment, this would just require extra work on my part in order to implement all the magic methods and such.

If history is to win again, then at least the documentation of the ABC  collections should clearly state the subtle, but IMHO important, differences between the two.
msg292475 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-04-27 19:57
For backward compatibility reasons, history will win.  

I'm not sure if documenting the additional restrictions of the concrete stdlib types in the ABC docs makes sense, either, since the ABCs are intentionally the minimal requirements for a concrete type to conform to the ABC.  The concrete ABC methods are really examples, even though they are also widely useful (and widely used).
msg292476 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2017-04-27 20:00
I suppose someone could propose a set of tests that an implementation of an ABC type should pass, to complement the stdlib type tests.  There would probably be a lot of bikeshedding involved in such tests though, so I don't hold out great hope anything would ever get accepted for merge :)
msg292507 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2017-04-28 03:54
> That being said, I believe (as the OP of issue 8743) 
> that the ABC collections should comply to the API of
> the Python base collections. 

Sorry Horacio, but your beliefs don't trump Guido's intentional design decisions.   He was the one who decided that the collections.abc.Set should have a __and__ that accepts an iterable and that it should not have the named methods (union, intersection, difference, etc).  Once released, that decision is not changeable without breaking existing code that relies on any iterable being accepted (which is a useful behavior given the absence of an union() method).   Guido also decided long ago that no class other that list would repeat its list.__iadd__ design mistake.   You can search the python-dev archives to find the discussion where it was decided that this applies to the concrete set type (i.e. that s |= 'abc' would raise a TypeError).  I happen to agree with that decision.  Though it seems to offend your sense of consistency, it does have the advantage of precluding what would likely be a common source of errors.

I'm marking this as closed.  The time to litigate API design decisions is prior to their being released.  This particular ship sailed a decade ago and has mostly worked out fine for all of our users.
History
Date User Action Args
2017-04-28 03:54:48rhettingersetstatus: open -> closed
resolution: not a bug
messages: + msg292507

stage: resolved
2017-04-27 20:00:28r.david.murraysetmessages: + msg292476
2017-04-27 19:57:59r.david.murraysetmessages: + msg292475
2017-04-27 17:59:16Horacio Hoyossetmessages: + msg292467
2017-04-26 07:22:33rhettingersetmessages: + msg292312
2017-04-24 04:03:45serhiy.storchakasetassignee: rhettinger
2017-04-23 21:14:23ned.deilysetnosy: + rhettinger, - ronaldoussoren
messages: + msg292186
components: - macOS
title: Fix for issue 8743 not available in python MacOS 3.6.1 -> Difference in behavior between set() and collections.abc.MutableSet() derived objects
2017-04-23 19:47:04Horacio Hoyossetfiles: + example.py

messages: + msg292182
2017-04-23 19:25:37ned.deilysetmessages: + msg292180
2017-04-23 19:00:24r.david.murraysetnosy: + r.david.murray
messages: + msg292178
2017-04-23 18:07:49Horacio Hoyoscreate