I posted a suggestion at python-ideas that the declaration of abstract properties could be improved in such a way that they could be declared with either the long-form or decorator syntax using the built-in property and abc.abstractmethod:

{{{
class MyProperty(property):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        for f in (self.fget, self.fset, self.fdel):
            if getattr(f, '__isabstractmethod__', False):
                self.__isabstractmethod__ = True
                break

class C(metaclass=ABCMeta):
    @MyProperty
    @abstractmethod
    def x(self):
        pass
    @x.setter
    @abstractmethod
    def x(self, val):
        pass

    # this syntax would also be supported:
    #@abstractmethod
    #def getx(self):
    #    pass
    #@abstractmethod
    #def setx(self, val):
    #    pass
    #x = MyProperty(getx, setx)

class D(C):
    'D does not define a concrete setter and cannot be instantiated'
    @C.x.setter
    def x(self):
        return 1

class E(D):
    'E has a concrete getter and setter, and can be instantiated'
    @D.x.setter
    def x(self, val):
        pass
}}}

It is hopefully evident that a relatively minor extension can be made to the built-in property such that @abstractproperty would no longer be needed. I have prepared a patch, complete with documentation and unit tests, but unfortunately I have not been able to test it because I have not been able to build Python from a mercurial checkout on either Ubuntu 11.04 or OS X 10.6.6 (for reasons unrelated to the patch.) BDFL thought the idea sounded good for inclusion in Python-3.3, and requested I submit the patch here.

The discussion on python-ideas: http://mail.python.org/pipermail/python-ideas/2011-March/009411.html

I looked at the patch (I didn't test it yet), my comments are on Rietveld.

Here is a new patch that addresses a couple problems found in review:

* Unit tests contained a typo (Property instead of property)
* DeprecationWarning would be issued when importing abc rather than when creating abstractproperty. (whether abstractproperty should be deprecated has been questioned).

I tried to test your patch, but the build dies with this error:
Fatal Python error: Py_Initialize: can't initialize sys standard streams
Traceback (most recent call last):
  File ".../cpython/Lib/io.py", line 60, in <module>
Aborted

I don't know why is this, but I get this error consistently with your patch, and no error without the patch.

On Sat, Mar 19, 2011 at 22:13, <dsdale24@gmail.com> wrote:
> Thank you for the feedback. The reason I suggested deprecating
> abstractproperty is that I think it is essentially broken. Subclasses
> have to redeclare the entire property, and if they forget to declare
> the setter for what is supposed to be a read/write property, there is 
> no way to catch it. With the new approach, it is possible to ensure
> that all the required features of the property have been implemented.
...
> On 2011/03/19 21:36:09, durban wrote:
> > I don't think abstractproperty should be deprecated. It is still
> > perfectly good to define a read-only abstract property (with one 
> > decorator instead of two).
>
> Zen of python.

I'm guessing you're referring to "There should be one-- and preferably only one --obvious way to do it."  That is a good point.  But currently the one way to:
- create an abstract static method: @abstractstaticmethod
- create an abstract class method: @abstractclassmethod
- create an abstract property: @abstractproperty (as you pointed out, this has some problems)

With your proposed change the one way to:
- create an abstract static method: @abstractstaticmethod
- create an abstract class method: @abstractclassmethod
- create an abstract property: @abstractmethod + @property
This is not a very good API.
Note, that a similar thing could be done for class/staticmethod, and then using @abstractmethod + @classmethod would be possible, and the API would be more consistent.  But it wasn't done because Guido objected it (see issue5867).

> This is the part where I am weak. Can you point me to documentation? 
> Why is an exception check necessary? Do PyObject_IsTrue and Py_DECREF 
> not know what to do when passed NULL?

http://docs.python.org/dev/py3k/c-api/object.html#PyObject_GetAttrString
If a Python API function returns NULL, that usually means that an exception was raised.  If you don't want the exception to propagate, you should call PyErr_Clear.  And I think it is not a good idea to call a function with NULL, unless the docs explicitly say that it can be passed NULL.

On Sun, Mar 20, 2011 at 5:18 AM, Daniel Urban <report@bugs.python.org> wrote:
>
> Daniel Urban <urban.dani+py@gmail.com> added the comment:
>
> I tried to test your patch, but the build dies with this error:
> Fatal Python error: Py_Initialize: can't initialize sys standard streams
> Traceback (most recent call last):
>  File ".../cpython/Lib/io.py", line 60, in <module>
> Aborted
>
> I don't know why is this, but I get this error consistently with your patch, and no error without the patch.
>
> On Sat, Mar 19, 2011 at 22:13, <dsdale24@gmail.com> wrote:
>> Thank you for the feedback. The reason I suggested deprecating
>> abstractproperty is that I think it is essentially broken. Subclasses
>> have to redeclare the entire property, and if they forget to declare
>> the setter for what is supposed to be a read/write property, there is
>> no way to catch it. With the new approach, it is possible to ensure
>> that all the required features of the property have been implemented.
> ...
>> On 2011/03/19 21:36:09, durban wrote:
>> > I don't think abstractproperty should be deprecated. It is still
>> > perfectly good to define a read-only abstract property (with one
>> > decorator instead of two).
>>
>> Zen of python.
>
> I'm guessing you're referring to "There should be one-- and preferably only one --obvious way to do it."  That is a good point.  But currently the one way to:
> - create an abstract static method: @abstractstaticmethod
> - create an abstract class method: @abstractclassmethod
> - create an abstract property: @abstractproperty (as you pointed out, this has some problems)
>
> With your proposed change the one way to:
> - create an abstract static method: @abstractstaticmethod
> - create an abstract class method: @abstractclassmethod
> - create an abstract property: @abstractmethod + @property
> This is not a very good API.

Unlike methods, properties are composite objects. It is therefore
reasonable that creating an abstract property might be a little
different from creating an abstract method.

> Note, that a similar thing could be done for class/staticmethod, and then using @abstractmethod + @classmethod would be possible, and the API would be more consistent.  But it wasn't done because Guido objected it (see issue5867).

Thank you for pointing that out. I've followed up with him at
python-ideas to seek clarification (he did not raise this point when I
posted the change to descrobject.c)

>> This is the part where I am weak. Can you point me to documentation?
>> Why is an exception check necessary? Do PyObject_IsTrue and Py_DECREF
>> not know what to do when passed NULL?
>
> http://docs.python.org/dev/py3k/c-api/object.html#PyObject_GetAttrString

I'm familiar with that page. Do you know of any documentation
addressing how to anticipate and respond to NULL?

I think a better idea would be to override getter and friends on the abstractproperty class.

On Sun, Mar 20, 2011 at 12:19 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> I think a better idea would be to override getter and friends on the abstractproperty class.

I just suggested the same at python-ideas. I'll work on an alternate patch.

On Sun, Mar 20, 2011 at 5:18 AM, Daniel Urban <report@bugs.python.org> wrote:
>
> Daniel Urban <urban.dani+py@gmail.com> added the comment:
>
> I tried to test your patch, but the build dies with this error:
> Fatal Python error: Py_Initialize: can't initialize sys standard streams
> Traceback (most recent call last):
>  File ".../cpython/Lib/io.py", line 60, in <module>
> Aborted
>
> I don't know why is this, but I get this error consistently with your patch, and no error without the patch.

Have you added any print statements to the patch? I'm working on a
completely new patch, which only touches abc.py on an existing
python3.2 install. When I add a print statement to the abstract
property creation routine, and run test_abc.py, I get the same error.

2011/3/20 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> On Sun, Mar 20, 2011 at 5:18 AM, Daniel Urban <report@bugs.python.org> wrote:
>>
>> Daniel Urban <urban.dani+py@gmail.com> added the comment:
>>
>> I tried to test your patch, but the build dies with this error:
>> Fatal Python error: Py_Initialize: can't initialize sys standard streams
>> Traceback (most recent call last):
>>  File ".../cpython/Lib/io.py", line 60, in <module>
>> Aborted
>>
>> I don't know why is this, but I get this error consistently with your patch, and no error without the patch.
>
> Have you added any print statements to the patch? I'm working on a
> completely new patch, which only touches abc.py on an existing
> python3.2 install. When I add a print statement to the abstract
> property creation routine, and run test_abc.py, I get the same error.

That's likely because the io library depends on abcs, so using print
in them creates a dependency cycle.

Thank you Daniel and Benjamin for the helpful feedback. I think the attached patch is a much better approach. It only touches abc.abstractproperty (instead of the builtin property), and uses a class method as a factory to return instances of either property or abstractproperty, depending on whether it holds any references to abstractmethods.

The patch is backwards compatible with the existing (though in my opinion, still broken) syntax of passing concrete methods to the abstractproperty constructor. I say that syntax is broken because properties are composite objects, and it is the methods that inherently make a property abstract or concrete. If one passes concrete getters and setters to abstractproperty, how do we know when the property has become concrete? Thus, I changed the documentation to refer to the more robust approach of passing abstractproperty methods that have been decorated with abstractmethod.

Unit tests are also provided. I still have not been able to build from my hg checkout, but I copied abc.py into my working 3.2 installation and ran the new test_abc.py without errors.

I'll be happy to address any additional concerns.

Here is a new version of the patch. I think it addresses all of the issues that have been raised to date.

I had to comment out the -lintl line in Modules/Setup to build on OS X, this seems to be a similar issue to http://bugs.python.org/issue6154 . So I don't have a _locale module, and I also don't have _scproxy. I ran "make test", and get the same results with and without the patch: 315 passes, 22 failed, 15 skipped. All of the failures are due to missing _locale and _scproxy, with the exception of an error during the sax test that is unrelated to my changes.

(Darren, what version of OS X and what arguments did you use for ./configure ?  In general, for testing purposes, a vanilla ./configure with no args should work fine for building a Python that works right from your source build directory.  If you want to build something to be installed, avoid using --enable-shared on OS X, see, for instance, Issue11445)

(Ned, I'm running 10.6.6 with a 64-bit kernel. I've tried running ./configure without any arguments, and also with --prefix=/opt/local, since I install essentially everything with MacPorts.)

(Darren, I'm not sure why you are running into problems with that setup.  I test with what sounds to be a very similar one including a MacPorts gettext port providing libintl although I do install ports as +universal (i386, x86_64) by default.  And I don't know why you would have problems with _scproxy.  If you would like to pursue, please open a separate issue with the results of your configure and make.)

I have an idea. How about instead of reusing abstractmethod for abstract getters and setters, you add abstractproperty.abstractgetter/setter/deleter?

Benjamin: have you thought this idea through?

2011/3/29 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> Benjamin: have you thought this idea through?

Perhaps inadequately?

I see some problems with this approach, but maybe I haven't fully appreciated it. Let me summarize the goals and constraints as I see them:

1) compatible with long-form and decorator syntax of {abstract}property declaration
2) backwards compatible, no change in semantics/behavior
3) decorator syntax needs to yield a concrete property once all abstract methods associated with the abstract property have been replaced with concrete implementations. (This is the reason why each abstract method associated with the property needs to get tagged with __isabstractmethod__. It provides an accounting of abstract methods associated with the property which fits with the existing semantics of abstract method declaration.)

The current approach actually satisfies all of the goals and constraints. It fits well with the existing semantics, there are no surprises and no changes in behavior for any existing code. It is even compatible with anyone who may have used @abstractmethod to decorate methods destined to be passed to @abstractproperty using the long-form property declaration (which would have worked even though it was not documented!)

The benefit of abstractproperty.abstract{...} is that one decorator is required instead of two, right? Are there others?

It is true that one could define abstract{getter,setter,deleter} decorators that would take care of setting the __isabstractmethod__ attribute on the function received, so that the @abstractmethod decorator would not be needed *once the property has been created*.

But if @abstractmethod is discouraged in favor of abstractproperty.abstractgetter and friends, abstractproperty would have to tag each method passed to its constructor as abstract (in order to support the long-form syntax and also the initial declaration with the decorator syntax) which would actually be a change in behavior with potential consequences. For example, maybe a third party defined a concrete getter in an abstract base class, and python-3.3 can't instantiate the subclasses because that getter was automatically tagged as abstract by the new abstractproperty constructor. So @abstractmethod would still be needed for methods passed to the constructor, meaning sometimes @abstractmethod would be needed, and sometimes it would not.

2011/3/29 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> I see some problems with this approach, but maybe I haven't fully appreciated it. Let me summarize the goals and constraints as I see them:
>
> 1) compatible with long-form and decorator syntax of {abstract}property declaration
> 2) backwards compatible, no change in semantics/behavior
> 3) decorator syntax needs to yield a concrete property once all abstract methods associated with the abstract property have been replaced with concrete implementations. (This is the reason why each abstract method associated with the property needs to get tagged with __isabstractmethod__. It provides an accounting of abstract methods associated with the property which fits with the existing semantics of abstract method declaration.)
>
> The current approach actually satisfies all of the goals and constraints. It fits well with the existing semantics, there are no surprises and no changes in behavior for any existing code. It is even compatible with anyone who may have used @abstractmethod to decorate methods destined to be passed to @abstractproperty using the long-form property declaration (which would have worked even though it was not documented!)
>
> The benefit of abstractproperty.abstract{...} is that one decorator is required instead of two, right? Are there others?

Mostly it doesn't create a weird asymmetry between a @abstractproperty
decorated function not needing @abstractmethod but
@someabstractprop.setter needing it.

>
> It is true that one could define abstract{getter,setter,deleter} decorators that would take care of setting the __isabstractmethod__ attribute on the function received, so that the @abstractmethod decorator would not be needed *once the property has been created*.
>
> But if @abstractmethod is discouraged in favor of abstractproperty.abstractgetter and friends, abstractproperty would have to tag each method passed to its constructor as abstract (in order to support the long-form syntax and also the initial declaration with the decorator syntax) which would actually be a change in behavior with potential consequences. For example, maybe a third party defined a concrete getter in an abstract base class, and python-3.3 can't instantiate the subclasses because that getter was automatically tagged as abstract by the new abstractproperty constructor. So @abstractmethod would still be needed for methods passed to the constructor, meaning sometimes @abstractmethod would be needed, and sometimes it would not.

That's not true. The method could be tagged in @abstractgetter decorator.

On Tue, Mar 29, 2011 at 9:31 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
> 2011/3/29 Darren Dale <report@bugs.python.org>:
>> The benefit of abstractproperty.abstract{...} is that one decorator is required instead of two, right? Are there others?
>
> Mostly it doesn't create a weird asymmetry between a @abstractproperty
> decorated function not needing @abstractmethod but
> @someabstractprop.setter needing it.

Did you read the documentation I provided in the patch? There is no
asymmetry, the documentation and examples provided by previous python
releases are demonstrably inadequate. For example:

class AbstractFoo(metaclass=ABCMeta):
    def get_bar(self): ...
    def set_bar(self, val): ...
    bar = abstractproperty(get_bar, set_bar)

The documentation indicates that a subclass will not be instantiable
until all of its abstract methods and properties are overridden. What
is abstract about the bar property? Was it the getter, setter, or
both, or neither? The answer is neither. A subclass can simply do:

class Foo(AbstractFoo):
    bar = property(AbstractFoo.get_bar, AbstractFoo.set_bar)

and it is instantiable. On the other hand, for AbstractFoo to assert
that subclasses must provide concrete implementations of the get_bar
and set_bar methods, it must decorate get_bar and set_bar with
@abstractproperty. This is true for previous releases of python, the
documentation of abstractproperty in previous python releases is
simply incomplete. If a method is abstract, it needs to have an
__isabstractmethod__ attribute that is True, and @abstractmethod
provides the means of setting this attribute.

This patch simply extends abstractproperty so it can respect the
abstractedness of the methods assigned to it. If somebody defines an
ambiguous abstractproperty like my AbstractFoo example, they get the
same result with the patch as they did without: an abstract property
with two concrete methods (this is an unfortunate situation that
cannot be fixed without breaking backwards compatibility).

Therefore, there is no asymmetry between when @abstractmethod is
required and when it is not. If the *method* is abstract and must be
reimplemented by a subclass, @abstractmethod is required. Even for
methods that participate in property definitions, even with
<=python-3.2.

>> It is true that one could define abstract{getter,setter,deleter} decorators that would take care of setting the __isabstractmethod__ attribute on the function received, so that the @abstractmethod decorator would not be needed *once the property has been created*.
>>
>> But if @abstractmethod is discouraged in favor of abstractproperty.abstractgetter and friends, abstractproperty would have to tag each method passed to its constructor as abstract (in order to support the long-form syntax and also the initial declaration with the decorator syntax) which would actually be a change in behavior with potential consequences. For example, maybe a third party defined a concrete getter in an abstract base class, and python-3.3 can't instantiate the subclasses because that getter was automatically tagged as abstract by the new abstractproperty constructor. So @abstractmethod would still be needed for methods passed to the constructor, meaning sometimes @abstractmethod would be needed, and sometimes it would not.
>
> That's not true. The method could be tagged in @abstractgetter decorator.

I think you misunderstood my point. I agreed with you that it could be
tagged by @abstractgetter. It cannot be tagged by the constructor.
That is where an asymmetry would be introduced between when
@abstractmethod is needed (declare methods abstract before passing
them to the constructor) and when it would not be (passing methods to
abstractgetter which declares them abstract).

(By the way, in review of issue11610.patch, GVR said he thought I had
the right idea and that the backward compatibility goal was satisfied.
Some of these points were covered in that discussion.)

Darren

On Tue, Mar 29, 2011 at 10:24 PM, Darren Dale <report@bugs.python.org> wrote:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> On Tue, Mar 29, 2011 at 9:31 PM, Benjamin Peterson
> <report@bugs.python.org> wrote:
>> 2011/3/29 Darren Dale <report@bugs.python.org>:
>>> The benefit of abstractproperty.abstract{...} is that one decorator is required instead of two, right? Are there others?
>>
>> Mostly it doesn't create a weird asymmetry between a @abstractproperty
>> decorated function not needing @abstractmethod but
>> @someabstractprop.setter needing it.
>
> Did you read the documentation I provided in the patch? There is no
> asymmetry, the documentation and examples provided by previous python
> releases are demonstrably inadequate. For example:
>
> class AbstractFoo(metaclass=ABCMeta):
>    def get_bar(self): ...
>    def set_bar(self, val): ...
>    bar = abstractproperty(get_bar, set_bar)
>
> The documentation indicates that a subclass will not be instantiable
> until all of its abstract methods and properties are overridden. What
> is abstract about the bar property? Was it the getter, setter, or
> both, or neither? The answer is neither. A subclass can simply do:
>
> class Foo(AbstractFoo):
>    bar = property(AbstractFoo.get_bar, AbstractFoo.set_bar)
>
> and it is instantiable. On the other hand, for AbstractFoo to assert
> that subclasses must provide concrete implementations of the get_bar
> and set_bar methods, it must decorate get_bar and set_bar with
> @abstractproperty.

Sorry, that should have read @abstractmethod.

So, are there objections to this patch, or can it be merged?

Is there anything preventing this patch from being merged?

2011/5/14 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> Is there anything preventing this patch from being merged?

I have to make time to think about the API a bit more.

On Sat, May 14, 2011 at 12:20 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>
>> Darren Dale <dsdale24@gmail.com> added the comment:
>>
>> Is there anything preventing this patch from being merged?
>
> I have to make time to think about the API a bit more.

Ok. Maybe you will come up with another alternative that hadn't
occurred to me. But I have given this issue quite a bit of thought,
considered several alternatives, and felt fortunate to find a solution
that preserves backwards compatibility, supports the property
decorator syntax, and meshes well with the existing syntax for
abstract methods. Perhaps, if you shared your concerns, I could help
address them and maybe save you some time.

Darrren

I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).

On Sat, May 14, 2011 at 4:28 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).

Right, but I explained why the redundancy is necessary in order to
preserve backwards compatibility. If the abstractproperty constructor
were changed to tag methods it receives as abstract, it would be a
backwards-incompatible change in behavior with potential consequences
for consumers of abstractproperty.
abstractproperty.abstract(getter/setter/deleter) could be implemented,
but it still wouldn't change the fact that if a getter/setter is
intended to be abstract, it needs to be decorated with @abstractmethod
before being passed to the abstractproperty() constructor. This is
true today in <=python-3.2: its not mentioned in the documentation,
but the behavior exists all the same.

Properties are composite objects, their behavior is defined by it is
the setters/getters/deleters they receive. So its actually a very
conceptually clean solution to decorate a method with @abstractmethod,
and it fits really nicely with the rest of the abc module. Why does
abstractproperty need special abstract(setter/getter/deleter) methods,
when the existing methods combine with @abstractmethod in a clean way
to produce the exact same result? To save one line of code?

Darren

2011/5/14 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> On Sat, May 14, 2011 at 4:28 PM, Benjamin Peterson
> <report@bugs.python.org> wrote:
>>
>> Benjamin Peterson <benjamin@python.org> added the comment:
>>
>> I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).
>
> Right, but I explained why the redundancy is necessary in order to
> preserve backwards compatibility. If the abstractproperty constructor
> were changed to tag methods it receives as abstract, it would be a
> backwards-incompatible change in behavior with potential consequences
> for consumers of abstractproperty.

I'm not suggesting that it tag methods it receives as abstract.
@getter/setter/deleter would still act the same.

> abstractproperty.abstract(getter/setter/deleter) could be implemented,
> but it still wouldn't change the fact that if a getter/setter is
> intended to be abstract, it needs to be decorated with @abstractmethod
> before being passed to the abstractproperty() constructor.

Why not? You could set the __abstractmethod__ attribute in abstractgetter().

> This is
> true today in <=python-3.2: its not mentioned in the documentation,
> but the behavior exists all the same.

>
> Properties are composite objects, their behavior is defined by it is
> the setters/getters/deleters they receive. So its actually a very
> conceptually clean solution to decorate a method with @abstractmethod,
> and it fits really nicely with the rest of the abc module. Why does
> abstractproperty need special abstract(setter/getter/deleter) methods,
> when the existing methods combine with @abstractmethod in a clean way
> to produce the exact same result? To save one line of code?

I find it produces a rather unfortunate ordering dependency for the
decorators which is hard to remember.

On Sat, May 14, 2011 at 5:17 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>
>> Darren Dale <dsdale24@gmail.com> added the comment:
>>
>> On Sat, May 14, 2011 at 4:28 PM, Benjamin Peterson
>> <report@bugs.python.org> wrote:
>>>
>>> Benjamin Peterson <benjamin@python.org> added the comment:
>>>
>>> I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).
>>
>> Right, but I explained why the redundancy is necessary in order to
>> preserve backwards compatibility. If the abstractproperty constructor
>> were changed to tag methods it receives as abstract, it would be a
>> backwards-incompatible change in behavior with potential consequences
>> for consumers of abstractproperty.
>
> I'm not suggesting that it tag methods it receives as abstract.
> @getter/setter/deleter would still act the same.

I wasn't talking about @getter/setter/deleter. I tried to be clear
that I was talking about the abstractproperty() constructor. It
doesn't currently tag the methods it receives as abstract, and to
change this would be a backward incompatible change. Therefore,
@abstractmethod should be used to tag methods as abstract before
passing them to the abstractproperty() constructor, and the abc
documentation should be changed to reflect this.

>> abstractproperty.abstract(getter/setter/deleter) could be implemented,
>> but it still wouldn't change the fact that if a getter/setter is
>> intended to be abstract, it needs to be decorated with @abstractmethod
>> before being passed to the abstractproperty() constructor.
>
> Why not? You could set the __abstractmethod__ attribute in abstractgetter().

I was not talking about decorating before passing @abstractgetter. I
was talking about decorating before passing to the abstractproperty()
constructor.

>> This is
>> true today in <=python-3.2: its not mentioned in the documentation,
>> but the behavior exists all the same.
>
>>
>> Properties are composite objects, their behavior is defined by it is
>> the setters/getters/deleters they receive. So its actually a very
>> conceptually clean solution to decorate a method with @abstractmethod,
>> and it fits really nicely with the rest of the abc module. Why does
>> abstractproperty need special abstract(setter/getter/deleter) methods,
>> when the existing methods combine with @abstractmethod in a clean way
>> to produce the exact same result? To save one line of code?
>
> I find it produces a rather unfortunate ordering dependency for the
> decorators which is hard to remember.

Why is it difficult to remember that you need to tag a method as
abstract before passing it to the property?

Okay: how about this. We retain the passing of @abstractmethod to
abstractpropert(), but @abstractgetter decorates the method for you.

2011/5/14 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> On Sat, May 14, 2011 at 5:17 PM, Benjamin Peterson
> <report@bugs.python.org> wrote:
>>
>> Benjamin Peterson <benjamin@python.org> added the comment:
>>
>> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>>
>>> Darren Dale <dsdale24@gmail.com> added the comment:
>>>
>>> On Sat, May 14, 2011 at 4:28 PM, Benjamin Peterson
>>> <report@bugs.python.org> wrote:
>>>>
>>>> Benjamin Peterson <benjamin@python.org> added the comment:
>>>>
>>>> I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).
>>>
>>> Right, but I explained why the redundancy is necessary in order to
>>> preserve backwards compatibility. If the abstractproperty constructor
>>> were changed to tag methods it receives as abstract, it would be a
>>> backwards-incompatible change in behavior with potential consequences
>>> for consumers of abstractproperty.
>>
>> I'm not suggesting that it tag methods it receives as abstract.
>> @getter/setter/deleter would still act the same.
>
> I wasn't talking about @getter/setter/deleter. I tried to be clear
> that I was talking about the abstractproperty() constructor. It
> doesn't currently tag the methods it receives as abstract, and to
> change this would be a backward incompatible change. Therefore,
> @abstractmethod should be used to tag methods as abstract before
> passing them to the abstractproperty() constructor, and the abc
> documentation should be changed to reflect this.
>
>>> abstractproperty.abstract(getter/setter/deleter) could be implemented,
>>> but it still wouldn't change the fact that if a getter/setter is
>>> intended to be abstract, it needs to be decorated with @abstractmethod
>>> before being passed to the abstractproperty() constructor.
>>
>> Why not? You could set the __abstractmethod__ attribute in abstractgetter().
>
> I was not talking about decorating before passing @abstractgetter. I
> was talking about decorating before passing to the abstractproperty()
> constructor.
>
>>> This is
>>> true today in <=python-3.2: its not mentioned in the documentation,
>>> but the behavior exists all the same.
>>
>>>
>>> Properties are composite objects, their behavior is defined by it is
>>> the setters/getters/deleters they receive. So its actually a very
>>> conceptually clean solution to decorate a method with @abstractmethod,
>>> and it fits really nicely with the rest of the abc module. Why does
>>> abstractproperty need special abstract(setter/getter/deleter) methods,
>>> when the existing methods combine with @abstractmethod in a clean way
>>> to produce the exact same result? To save one line of code?
>>
>> I find it produces a rather unfortunate ordering dependency for the
>> decorators which is hard to remember.
>
> Why is it difficult to remember that you need to tag a method as
> abstract before passing it to the property?

I don't think the common case should be passing things to
abstractproperty(), rather using the decorator.

On Sat, May 14, 2011 at 5:55 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> Okay: how about this. We retain the passing of @abstractmethod to
> abstractpropert(), but @abstractgetter decorates the method for you.

That can work, although I would advise against it. I find it strange
that we would use @abstractmethod sometimes and not others. If that is
what it takes to get the patch accepted, so be it. But since I don't
understand the motivation behind this approach, I won't be the one to
document the special cases of when @abstractmethod is required and
when it is not.

> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>
>> Darren Dale <dsdale24@gmail.com> added the comment:
>>
>> On Sat, May 14, 2011 at 5:17 PM, Benjamin Peterson
>> <report@bugs.python.org> wrote:
>>>
>>> Benjamin Peterson <benjamin@python.org> added the comment:
>>>
>>> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>>>
>>>> Darren Dale <dsdale24@gmail.com> added the comment:
>>>>
>>>> On Sat, May 14, 2011 at 4:28 PM, Benjamin Peterson
>>>> <report@bugs.python.org> wrote:
>>>>>
>>>>> Benjamin Peterson <benjamin@python.org> added the comment:
>>>>>
>>>>> I still dislike the reduntancy of having abstractmethod and abstractproperty on a method. I think a better idea is having abstractproperty.abstract(getter/setter/deleter).
>>>>
>>>> Right, but I explained why the redundancy is necessary in order to
>>>> preserve backwards compatibility. If the abstractproperty constructor
>>>> were changed to tag methods it receives as abstract, it would be a
>>>> backwards-incompatible change in behavior with potential consequences
>>>> for consumers of abstractproperty.
>>>
>>> I'm not suggesting that it tag methods it receives as abstract.
>>> @getter/setter/deleter would still act the same.
>>
>> I wasn't talking about @getter/setter/deleter. I tried to be clear
>> that I was talking about the abstractproperty() constructor. It
>> doesn't currently tag the methods it receives as abstract, and to
>> change this would be a backward incompatible change. Therefore,
>> @abstractmethod should be used to tag methods as abstract before
>> passing them to the abstractproperty() constructor, and the abc
>> documentation should be changed to reflect this.
>>
>>>> abstractproperty.abstract(getter/setter/deleter) could be implemented,
>>>> but it still wouldn't change the fact that if a getter/setter is
>>>> intended to be abstract, it needs to be decorated with @abstractmethod
>>>> before being passed to the abstractproperty() constructor.
>>>
>>> Why not? You could set the __abstractmethod__ attribute in abstractgetter().
>>
>> I was not talking about decorating before passing @abstractgetter. I
>> was talking about decorating before passing to the abstractproperty()
>> constructor.
>>
>>>> This is
>>>> true today in <=python-3.2: its not mentioned in the documentation,
>>>> but the behavior exists all the same.
>>>
>>>>
>>>> Properties are composite objects, their behavior is defined by it is
>>>> the setters/getters/deleters they receive. So its actually a very
>>>> conceptually clean solution to decorate a method with @abstractmethod,
>>>> and it fits really nicely with the rest of the abc module. Why does
>>>> abstractproperty need special abstract(setter/getter/deleter) methods,
>>>> when the existing methods combine with @abstractmethod in a clean way
>>>> to produce the exact same result? To save one line of code?
>>>
>>> I find it produces a rather unfortunate ordering dependency for the
>>> decorators which is hard to remember.
>>
>> Why is it difficult to remember that you need to tag a method as
>> abstract before passing it to the property?
>
> I don't think the common case should be passing things to
> abstractproperty(), rather using the decorator.

It definitely is a common case, and always will be. You can't begin
using abstractproperty.abstract(getter/setter/deleter) until you have
an abstract property, which requires passing a (potentially abstract)
method to the constructor.

2011/5/14 Darren Dale <report@bugs.python.org>:
>
> Darren Dale <dsdale24@gmail.com> added the comment:
>
> On Sat, May 14, 2011 at 5:55 PM, Benjamin Peterson
> <report@bugs.python.org> wrote:
>>
>> Benjamin Peterson <benjamin@python.org> added the comment:
>>
>> Okay: how about this. We retain the passing of @abstractmethod to
>> abstractpropert(), but @abstractgetter decorates the method for you.
>
> That can work, although I would advise against it. I find it strange
> that we would use @abstractmethod sometimes and not others. If that is
> what it takes to get the patch accepted, so be it. But since I don't
> understand the motivation behind this approach, I won't be the one to
> document the special cases of when @abstractmethod is required and
> when it is not.

It would just be a convenience.

> It definitely is a common case, and always will be. You can't begin
> using abstractproperty.abstract(getter/setter/deleter) until you have
> an abstract property, which requires passing a (potentially abstract)
> method to the constructor.

What about

@abstractproperty
def something(): pass

@abstractproperty.setter
def set(): pass

@abstractproperty.deleter
def delete: pass

requires you to pass a method (explicitly) to a constructor?

On Sat, May 14, 2011 at 6:24 PM, Benjamin Peterson
<report@bugs.python.org> wrote:
>
> Benjamin Peterson <benjamin@python.org> added the comment:
>
> 2011/5/14 Darren Dale <report@bugs.python.org>:
>>
>> Darren Dale <dsdale24@gmail.com> added the comment:
>>
>> It definitely is a common case, and always will be. You can't begin
>> using abstractproperty.abstract(getter/setter/deleter) until you have
>> an abstract property, which requires passing a (potentially abstract)
>> method to the constructor.
>
> What about
>
> @abstractproperty
> def something(): pass
>
> @abstractproperty.setter
> def set(): pass
>
> @abstractproperty.deleter
> def delete: pass
>
> requires you to pass a method (explicitly) to a constructor?

@abstractproperty
def something(): pass

takes the "something" function and passes it to the abstractproperty()
constructor.

It doesn't appear that you are familiar with how the decorator syntax
works for properties. Here is how your example should probably look:

@abstractproperty
def something(): pass

@something.setter
def something(): pass

@something.deleter
def something(): pass

It just occurred to me, there is a potential problem with abstractproperty and the decorator syntax in my patch:

class Foo:

    @abstractproperty
    def p(self): pass
    # p is abstract, but has no abstract methods

    @p.setter
    def p(self, val): pass
    # p has no abstract properties, at this point it becomes an instance
    # of property, not abstractproperty

    @p.deleter
    @abstractmethod
    def p(self): pass
    # the deleter was tagged as abstract, but p was already a
    # regular property. There is no way to turn a regular
    # property into an abstractproperty, so the abstractedness
    # of the deleter is not respected.

Really, the ideal approach is the original one: provide the builtin property with an __isabstractmethod__ attribute, which is set to True if the property has any abstract methods, and False otherwise. (My C is probably too weak to modify the property builtin on my own).

FWIW, if you still want to advance this, you could bring it up on Python-dev. I still feel uncomfortable with the API but could be convinced with others thought it was the best solution.

Thank you for the suggestion. I will follow up at python-dev, but it will probably be a few weeks before I have time to do a proper job of it.

I posted the following at python-dev (http://mail.python.org/pipermail/python-dev/2011-June/111837.html):

I would like to try to address some shortfalls with the way python deals with
abstract base classes containing descriptors. I originally was just concerned
with improving support for defining abstract properties with the decorator
syntax and converting between abstract and concrete properties, but recently
realized that the problem extends to descriptors in general.

ABCs
----

First, a bit of background may be in order. An abstract base class is defined
by specifying its metaclass as ABCMeta (or a subclass thereof)::

   class MyABC(metaclass=ABCMeta):
       @abstractmethod
       def foo(self):
           pass

When trying to instantiate MyABC or any of its subclasses, ABCMeta inspects the
current class namespace for items tagged with __isabstractmethod__=True::

   class ABCMeta(type):
   #[...]
       def __new__(mcls, name, bases, namespace):
           cls = super().__new__(mcls, name, bases, namespace)
           # Compute set of abstract method names
           abstracts = {name
                        for name, value in namespace.items()
                        if getattr(value, "__isabstractmethod__", False)}

ABCMeta then checks if any of the base classes define any items tagged with
__isabstractmethod__ and whether they remain abstract in the current
class namespace::

           for base in bases:
               for name in getattr(base, "__abstractmethods__", set()):
                   value = getattr(cls, name, None)
                   if getattr(value, "__isabstractmethod__", False):
                       abstracts.add(name)
           cls.__abstractmethods__ = frozenset(abstracts)

In Objects/typeobject.c, __abstractmethods__ is actually a descriptor, and
setting it gives the type a chance to set an internal flag specifying if it
has any abstract methods defined. When object_new is called in typeobject.c,
the flag is checked and an error is raised if any abstract methods were
identified.

Issues with ABCs and descriptors
--------------------------------

In order for this scheme to work, ABCMeta needs to identify all of the abstract
methods, but there are some limitations when we consider descriptors. For
example, Python's property is a composite object, whose behavior is defined by
the getter, setter, and deleter methods with which it is composed. Since there
is already an @abstractmethod decorator, I would have suspected that defining
abstract properties would be intuitive::

   class MyABC(metaclass=ABCMeta):
       @abstractmethod
       def _get_foo(self):
           pass
       @abstractmethod
       def _set_foo(self, val):
           pass
       foo = property(_get_foo, _set_foo)

       @property
       @abstractmethod
       def bar(self):
           pass
       @bar.setter
       @abstractmethod
       def bar(self, val):
           pass

Ideally, one would want the flexibility of defining a concrete getter and an
abstract setter, for example. However, ABCMeta does not inspect the descriptors
of a class to see if they contain any abstract methods. It only inspects the
descriptor itself for a True __isabstractmethod__ attribute. This places the
burdon on every descriptor implementation to provide its own support for ABC
compatibility. For example, support for abstract properties was attempted by
adding abstractproperty to the abc module. abstractproperty subclasses the
property builtin (as opposed to the relationship between every other abstract
and concrete class in the python language). Here is the definition of
abstractproperty, in its entirety (modulo docstrings)::

   class abstractproperty(property):
       __isabstractmethod__ = True

A number of problems manifest with this approach, and I think they all can be
traced to the fact that the abstractedness of a descriptor is currently not
dependent upon the abstractedness of the methods with which it is
composed. The documentation for abstractproperty doesn't suggest using
@abstractmethod::

       class C(metaclass=ABCMeta):
           def getx(self): ...
           def setx(self, value): ...
           x = abstractproperty(getx, setx)

which leads to Issue #1: What is abstract about C.x? How does a subclass of C
know whether it needs to override the getter or setter?

Issue #2: The decorator syntax cannot be used to convert an abstract property
into a concrete one. (This relates to Issue #1: how would a descriptor even know
when such a conversion would be appropriate?) Running the following code::

   from abc import ABCMeta, abstractmethod, abstractproperty

   class AbstractFoo(metaclass=ABCMeta):
       @abstractproperty
       def bar(self):
           return 1
       @bar.setter
       def bar(self, val):
           pass

   class ConcreteFoo(AbstractFoo):
       @AbstractFoo.bar.getter
       def bar(self):
           return 1
       @bar.setter
       def bar(self, val):
           pass
   foo = ConcreteFoo()

yields::

   TypeError: Can't instantiate abstract class ConcreteFoo with abstract
   methods bar

Issue #3: The following class *is* instantiable, even though
AbstractFoo declared that a setter for bar is required::

   class ConcreteFoo(AbstractFoo):
       @property
       def bar(self):
           pass

Previous attempt to improve abc.abstractproperty
------------------------------------------------

It seems to me that the strategy used by abc.abstractproperty is fundamentally
ill-advised. I explored the possibility of extending abstractproperty,
redefining its getter, setter, and deleter methods such that they would work in
conjunction with the @abstractmethod decorator and yield an instance of the
builtin property once all abstract methods were replaced with concrete ones
(http://bugs.python.org/issue11610). Issues #1 and #2 were addressed, but there
were still problems with that approach. It did not address Issue #3, and it
also introduced a new issue, #4::

   class AbstractFoo(metaclass=ABCMeta):
       @abstractproperty
       # bar would be an abstractproperty, even though the getter is concrete
       def bar(self):
           return 1
       @bar.setter
       # bar.setter inspected the getter and the new setter, did not identify
       # any abstract methods, and thus returned an instance of the built-in
       # property
       def bar(self, val):
           pass
       @bar.deleter
       # bar is a concrete property, its deleter decorator does not know it
       # is supposed to check for abstract methods, so it will return an
       # instance of the built-in property:
       @abstractmethod
       def bar(self):
           pass

By the time the deleter was specified, bar was a concrete property, which does
not know it should return an instance of abstractproperty (in part because the
inheritance diagram for property/abstractproperty is inverted). Thus,
AbstractFoo was instantiable, even though it shouldn't be.

Finally, issue #5: the current approach taken by ABCMeta and abstractproperty
places the burdon on descriptors to identify themselves to ABCMeta as
abstract. Considering the issues encountered with abstractproperty, this may be
an onerous requirement.

There has been a fair amount of discussion at http://bugs.python.org/issue11610
, which can be summarized as a) concerns about maintaining backward
compatibility, and b) objections to requiring @abstractmethod to specify that a
method being passed to abstractproperty is abstract.

Extending ABCMeta: A Promising Way Forward
------------------------------------------

I think the key is to focus on Issue #3. ABCMeta needs to be improved to
recognize descriptor objects, both in the current namespace as well as the base
classes, and to identify any abstract methods associated with the
descriptors. I suggest the following approach in ABCMeta::

   def __new__(mcls, name, bases, namespace):
       cls = super().__new__(mcls, name, bases, namespace)
       # Compute set of abstract method names

       def isdescriptor(val):
           return hasattr(val, '__get__') or hasattr(val, '__set__') \
                   or hasattr(val, '__delete__')
       def getabstracts(ns):
           return [name
                   for name, value in ns.items()
                   if getattr(value, "__isabstractmethod__", False)]

       abstracts = getabstracts(namespace)
       for item, val in namespace.items():
           # further inspect descriptors for abstract methods:
           if isdescriptor(val):
               ## unfortunately, can't import inspect:
               #from inspect import getmembers
               #d = dict(getmembers(val))
               ## so instead, use the following:
               d = dict((k, getattr(val, k, None)) for k in dir(val))
               for name in getabstracts(d):
                   # add the abstract descriptor methods to the list:
                   abstracts.append('%s.%s'%(item, name))
       for base in bases:
           for name in getattr(base, "__abstractmethods__", set()):
               if '.' in name:
                   # base class identified a descriptor abstract method:
                   k, v = name.split('.')
                   desc = getattr(cls, k, None)
                   val = getattr(desc, v, None)
               else:
                   val = getattr(cls, name, None)
               if val is None or getattr(val, "__isabstractmethod__", False):
                   abstracts.append(name)
       cls.__abstractmethods__ = frozenset(abstracts)

I rolled everything into __new__ just to keep it as simple as possible for the
sake of discussion. Python already provides the rest of the framework needed
for descriptors to work properly with ABCs. This implementation actually works;
I've tested it with an existing python-3.2 install::

   from abc import ABCMeta, abstractmethod

   class AbstractFoo(metaclass=ABCMeta):
       @property
       @abstractmethod
       def bar(self):
           return 1
       @bar.setter
       @abstractmethod
       def bar(self, val):
           pass

   >>> abstractfoo = AbstractFoo()
   Traceback (most recent call last):
     File "temp.py", line 17, in <module>
       abstractfoo = AbstractFoo()
   TypeError: Can't instantiate abstract class AbstractFoo with abstract
   methods bar.fget, bar.fset

as expected. Note the more informative error message indicating what about the
bar property is abstract. Also::

   class ConcreteFoo(AbstractFoo):
       @AbstractFoo.bar.getter
       def bar(self):
           return 1

   >>> foo = ConcreteFoo()
   Traceback (most recent call last):
     File "temp.py", line 24, in <module>
       foo = ConcreteFoo()
   TypeError: Can't instantiate abstract class ConcreteFoo with abstract
   methods bar.fset

So issue #1 is addressed, since we are explicitly specifying which descriptor
methods are abstract. Issue #2 has been addressed, since the following class is
instantiable::

   class ConcreteFoo(AbstractFoo):
       @AbstractFoo.bar.getter
       def bar(self):
           return 1
       @bar.setter
       def bar(self, val):
           pass

Issue #3 is also addressed. In the following example, even though I redefine
the bar property as a readonly property, ABCMeta recognizes that a setter is
needed::

   class ConcreteFoo(AbstractFoo):
       @property
       def bar(self):
           return 1

   >>> foo = ConcreteFoo()
   Traceback (most recent call last):
     File "temp.py", line 24, in <module>
       foo = ConcreteFoo()
   TypeError: Can't instantiate abstract class ConcreteFoo with abstract
   methods bar.fset

Issue #4 (introduced in a previous attempt to solve the problem
using abstractproperty) is also addressed::

   class AbstractFoo(metaclass=ABCMeta):
       @property
       def bar(self):
           return 1
       @bar.setter
       def bar(self, val):
           pass
       @bar.deleter
       @abstractmethod
       def bar(self):
           pass

   >>> abstractfoo = AbstractFoo()
   Traceback (most recent call last):
     File "temp.py", line 15, in <module>
       abstractfoo = AbstractFoo()
   TypeError: Can't instantiate abstract class AbstractFoo with abstract
   methods bar.fdel

Finally, this approach addresses Issue #5 by holding ABCMeta responsible for
identifying the abstractedness of descriptor methods, rather than placing that
burdon on the desciptor objects to identify themselves as abstract. If ABCMeta
were extended as above to identify abstract methods associated with
descriptors, third parties would simply decorate methods used to compose the
descriptors with @abstractmethod.

This change to ABCMeta would not effect the behavior of abstractproperty, so
backward compatibility would be maintained in that respect. But I think
abstractproperty should be deprecated, or at the very least removed from the
documentation. The documentation for @abstractmethod in >=python-3.3 should be
extended to provide examples with properties/descriptors. The syntax would be
backward compatible with older python versions, but with <python-3.3 ABCMeta
would simply not recognize descriptors' abstract methods. This leads to one
source of potential forward compatibility::

   class AbstractFoo(metaclass=ABCMeta):
       @property
       @abstractmethod
       def bar(self):
           return 1

   class ConcreteFoo(AbstractFoo):
       pass

Both above classes would be instantiable with <python-3.3, but not with
>=python3.3. In my opinion, this is a feature: python-3.3 has identified a bug
in ConcreteFoo. The developer would not have tagged that method as abstract
unless they had intended for consumers of AbstractFoo to provide a concrete
implementation.

Here is an improved patch, which includes feedback from python-dev. "make test" runs without failures, however test_abc.py prints deprecation warnings for abstractproperty. I'm not familiar with the protocol here, do we continue to include unit tests for deprecated features until they are removed?

It doesn't work with staticmethod:

>>> import abc
>>> 
>>> class C(metaclass=abc.ABCMeta):
...     @staticmethod
...     @abc.abstractmethod
...     def foo(x):
...             raise NotImplementedError()
... 
>>> class D(C):
...     @staticmethod
...     def foo(x):
...             return x + 1
... 
>>> D()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class D with abstract methods foo.__func__
>>>

On Sat, Jun 11, 2011 at 3:11 AM, Daniel Urban <report@bugs.python.org> wrote:
>
> Daniel Urban <urban.dani+py@gmail.com> added the comment:
>
> It doesn't work with staticmethod:
>
>>>> import abc
>>>>
>>>> class C(metaclass=abc.ABCMeta):
> ...     @staticmethod
> ...     @abc.abstractmethod
> ...     def foo(x):
> ...             raise NotImplementedError()
> ...
>>>> class D(C):
> ...     @staticmethod
> ...     def foo(x):
> ...             return x + 1
> ...
>>>> D()
> Traceback (most recent call last):
>  File "<stdin>", line 1, in <module>
> TypeError: Can't instantiate abstract class D with abstract methods foo.__func__

You still need to use @abc.abstractstaticmethod.

[...]
>> Traceback (most recent call last):
>>  File "<stdin>", line 1, in <module>
>> TypeError: Can't instantiate abstract class D with abstract methods foo.__func__
>
> You still need to use @abc.abstractstaticmethod.

Thinking about this some more, I think the error you found should be
considered a problem. The issue is with the descriptor access itself:
In class C we inspect the staticmethod instance in the namespace dict
passed to ABCMeta and find foo's abstract __func__. But later, ABCMeta
identifies C as a base of D, identifies C.foo.__func__ from
C.__abstractmethods__, then does a getattr on the formative D class.
D.__dict__['foo'] is a descriptor, which when accessed as D.foo
returns D.__dict__['foo'].__func__ (as opposed to the other
descriptors we have been discussing, where descriptor "get" access is
only invoked by an instance of the class). ABCMeta needs to inspect
the descriptor itself, not whatever the descriptor wants to return
when accessed. We can't use D.__dict__ to access the foo descriptor,
since some other base class of D may have provided the concrete
implementation of foo. Does anyone know another way to search the MRO
and return the foo descriptor?

You helped bring up another point: all functions are descriptors,
since they all have __get__ methods. That means every method is going
to be inspected for abstract members. Is this a problem?

inspect.getattr_static has the necessary logic to search for descriptors without invoking them.

However, it may be better to revert to the idea of pushing this functionality back onto the individual descriptors and have the problematic descriptors like property and staticmethod simply implement __isabstractmethod__ as a property.

property:
  @property
  def __isabstractmethod__(self):
    return (self.fget.__isabstractmethod__ or
            self.fset.__isabstractmethod__ or
            self.fdel.__isabstractmethod__)

staticmethod/classmethod:

  @property
  def __isabstractmethod__(self):
    return self.__func__.__isabstractmethod__

With this approach, the "one true way" to handle abstract descriptors would be to do:

  #instance method
  @abstractmethod
  def f(self):
    ...

  @property
  @abstractmethod
  def f(self):
    ...

  @classmethod
  @abstractmethod
  def f(self):
    ...

  @staticmethod
  @abstractmethod
  def f(self):
    ...

This wouldn't allow for the prettier error messages, but it's much cleaner than having ABCMeta trawling through class attribute dir() lists.

On Sat, Jun 11, 2011 at 8:55 AM, Nick Coghlan <report@bugs.python.org> wrote:
>
> Nick Coghlan <ncoghlan@gmail.com> added the comment:
>
> inspect.getattr_static has the necessary logic to search for descriptors without invoking them.

Unfortunately, we can't import inspect, even inside ABCMeta.__new__.

> However, it may be better to revert to the idea of pushing this functionality back onto the individual descriptors and have the problematic descriptors like property and staticmethod simply implement __isabstractmethod__ as a property.
>
> property:
>  @property
>  def __isabstractmethod__(self):
>    return (self.fget.__isabstractmethod__ or
>            self.fset.__isabstractmethod__ or
>            self.fdel.__isabstractmethod__)
>
> staticmethod/classmethod:
>
>  @property
>  def __isabstractmethod__(self):
>    return self.__func__.__isabstractmethod__

That's a good idea.

> With this approach, the "one true way" to handle abstract descriptors would be to do:
>
>  #instance method
>  @abstractmethod
>  def f(self):
>    ...
>
>  @property
>  @abstractmethod
>  def f(self):
>    ...
>
>  @classmethod
>  @abstractmethod
>  def f(self):
>    ...
>
>  @staticmethod
>  @abstractmethod
>  def f(self):
>    ...
>
> This wouldn't allow for the prettier error messages, but it's much cleaner than having ABCMeta trawling through class attribute dir() lists.

Those classes could conceivably do:

@property
def __abstractmethods__(self):
    return ("...", ...)

It would not be required, but if ABCMeta found it, it could use it.
Just a thought.

Darren

[...]
>
> This wouldn't allow for the prettier error messages, but it's much cleaner than having ABCMeta trawling through class attribute dir() lists.

I think there is another reason to do it this way. Suppose I have a
custom descriptor MyProperty that stores its getter as
MyProperty.my_fget. In this example:

class C:
    @property
    @abstractmethod
    def foo(self): return 1

class D:
    @MyProperty
    def foo(self): return 2

if ABCMeta determines C.foo.fget is abstract, it will not recognize
that D.foo.my_fget supplies the concrete implementation. MyProperty
may provide the same interface as property, it may even subclass
property, but there is no guarantee its implementation conforms to
property the way ABCMeta is expecting it to.

The downside to making __isabstractmethod__ a property is that if C
declares foo to be a read/write property, and D redefines foo as
read-only and concrete, ABCMeta will not recognize the oversight. That
seems a significant deficiency to me. I'm not sure how to proceed.

Darren

Perhaps rather than changing ABCMeta, provide a base descriptor class that has __isabstractmethod__ implemented to calculate the abstractness.  Then property could use that, as could any of the other relevant descriptors we have around.  The __isabstractmethod__ attribute of the descriptor would itself need to be a data-descriptor (which property() is).  That would ensure that __isabstractmethod__ is not set directly on the descriptor instance.

I agree with Nick that it may be better to have the descriptor classes take care of managing __isabstractmethod__ themselves, instead of having ABCMeta compute it.  Special-casing ABCMeta to handle custom __isabstractmethod__ calculation for any specific type seems like something we should avoid.

Per your last message, if a specific descriptor has an abstract setter then the descriptor should be considered abstract.  If the implementation of that attribute is not a descriptor should it raise a TypeError?  If it is a descriptor but it does not have a setter, should it likewise fail?  I'm not so sure.  We already don't enforce types on abstract attributes.  You don't have to implement an abstractproperty with a property, an abstractstaticmethod with a staticmethod, nor an abstractclassmethod with a classmethod.  For that matter you don't have to implement an abstractmethod with a function.  It just has to be an object bound to the same name.  Should descriptors get special treatment over any other type that implements __isabstractmethod__?

That brings up a concern of mine.  I've found the name abstractmethod (specifically the "method" part) to be misleading.  Any attribute can be "abstract" and ABCMeta only cares about __isabstractmethod__.  Maybe having "method" in the name is meant to imply the expected use case?  I would rather they were called  __isabstractattribute__, and __abstractattributes__, which would be less confusing when using ABCs, particularly at first.  Having "attribute" in the name is nice, since it makes it clear we're talking about attributes, rather than other abstraction contexts.  Having a specific abstractmethod decorator is still good since we only decorate functions in an ABC.

I'm +1 for deprecating abstractproperty and really all the decorators except abstractmethod (if the use cases were addressed).  If abstractmethod were smart about on which object it sets __isabstractmethod__, the other decorators would be unnecessary;  and the order in which you decorate would not matter as much.  The other decorators could become simple wrappers around abstractmethod if we felt the need to keep them around.  It's nice that the decorators say with what you are expecting to replace them, but you can get that by using the corresponding normal decorators.

Remember the goal here is *not* to completely eliminate the need to test that objects implement an ABC correctly. It's to make it easier to declare the expected interface in a way that helps readers of the ABC definition to figure out what is going on, and to reduce the proliferation of abstract descriptors.

Since we made a deliberate choice not to do signature checks when ABCs were added to the language, there's nothing an ABC can do to stop someone (for example) overriding an abstract method or descriptor "foo" with "foo = 1". That's almost certainly wrong, but it's wrong at a level that ABCs don't care about.

If someone incorrectly overrides a read/write property with a read-only property and there's nothing in their test suite that picks that up, then that's a flaw in the test suite, not a flaw in the ABC itself.

Regarding the naming, @abstractmethod and __isabstractmethod__ are definitely about methods, or method-based descriptors (such as property). There *could* be a case to be made to allow for abstract data attributes, but really, using an abstract property should cover that use case well enough.

Didn't mean to sidetrack.  :)  I really appreciate the effort Darren has put into this.

> there's nothing an ABC can do to stop someone (for example) overriding
> an abstract method or descriptor "foo" with "foo = 1".

I’ve find it useful to use an abstractproperty to specify an attribute that concrete subclasses have to define.  Was that wrong?  From a technical viewpoint, I replaced a method with a data attribute, but from a doc/human viewpoint, replacing a property with a regular attribute did not seem wrong to me.

So, if there are guidelines about “almost certainly wrong” uses of the ABC machinery, they should IMO be documented.

In that paragraph, I was only talking about cases where "foo = 1" *isn't* a valid override (which, I hope you'll agree, it typically won't be).

Your described approach of declaring an abstract property and then overriding it with an ordinary class attribute is part of the answer I gave Eric in pointing out why a separate concept of an abstract attribute isn't really necessary.

On Sat, Jun 11, 2011 at 7:32 PM, Eric Snow <report@bugs.python.org> wrote:
>
> Eric Snow <ericsnowcurrently@gmail.com> added the comment:
> Per your last message, if a specific descriptor has an abstract setter then the descriptor should be considered abstract.  If the implementation of that attribute is not a descriptor should it raise a TypeError?  If it is a descriptor but it does not have a setter, should it likewise fail?

Consider a framework like Enthought's Traits or Riverbank Computing's
dip, where setting the value of a descriptor can result in other
objects being notified of the change. Both Traits and dip are based on
the concept of interfaces, but imagine someone wanted to develop
something similar based on ABCs. In that case, one could argue that
replacing the descriptor with a regular attribute, or another
read-only descriptor, would not satisfy the ABC specification. Then it
might be nice if the abc mechanisms could catch the error. But it
looks like this will be difficult in cases where the subclasses
replaces the descriptor, unless perhaps an AbstractDescriptor were
provided that explained how ABCMeta is going to identify abstract
methods:

    class AbstractDescriptor(metaclass=abc.ABCMeta):
        @property
        @abc.abstractmethod
        def __abstractmethods__(self):
            # it would be nice if descriptors new their own names here,
            # __abstractmethods__ could return: ('bar.fget', 'bar.fset')
            return frozenset(m for m in ('fget', 'fset', 'fdel')
                             if getattr(getattr(self, m, None),
                                        '__isabstractmethod__', False))
        @property
        def __isabstractmethod__(self):
            return True if self.__abstractmethods__

    AbstractDescriptor.register(property)

Of course, not all descriptors would be required to derive from
AbstractDescriptor. There is no intended stick, but the carrot is
better integration with with ABCs.

Having said all that, I think the above suggestion including
__abstractmethods__ for descriptors makes unreasonable demands of
conformity between various descriptor implementations, and that Nick's
suggestion of simply asking descriptors to provide an
__isabstractmethod__ descriptor is probably good enough. Sufficient
documentation of an ABC's interface can cover the rest.

The inspect module or something like it may still be needed in ABCMeta
to work around the general issue Daniel discovered with staticmethod.
That way ABCMeta could inspect the descriptors themselves and attempt
to retrieve their __isabstractmethod__ value.

(aside: unlike ABCMeta.__new__, ABCMeta.register makes no attempt to
verify that the class being registered actually meets the
specification. Why not have ABCMeta.register perform the same checks
as ABCMeta.__new__, and raise an error if the registered class does
not conform to the specification?)

My work is going to keep me pretty busy over the next three weeks, and
I'm still not accomplished with Python's C-API. If someone else wants
to take a crack at the next patch, please feel free.

Darren

Non-conformant explicit registration is permitted on purpose to allow developers to only supply partial implementations when it is known that that is all a given application requires. Extremely impure, but quite practical :)

Note that the core logic of inspect.getattr_static isn't all that complicated. If necessary, it could be moved inside the module with ABCMeta and then wrapped or reference by the inspect module.

Here is attempt #4. This patch extends the property, classmethod and staticmethod builtins with an __isabstractmethod__ descriptor. Docs and tests are updated as well. "make test" runs without failures. This is my first real attempt with the C-API, and I think I am finally over the learning curve, but comments would be greatly appreciated.

I've posted some comments on Rietveld.

I've requested additional feedback based on comments at Rietveld.

Here is a new version of the patch, addressing points raised in the review of the previous version.

Any additional comments?

It would be nice if this patch could be merged in time for python-3.3...

I noticed this in an older message:

> test_abc.py prints deprecation warnings for abstractproperty. I'm not
> familiar with the protocol here, do we continue to include unit tests
> for deprecated features until they are removed?
We try to keep the test warning-free.  If your patch deprecates something, the tests should not use deprecated API, except in one method that tests the deprecation itself.  There are helpers in test.support and warnings to catch and inspect warnings.

3.3 is still months away; there’s time for your patch.  If nobody replies here, ping python-dev again.

I just double-checked that the unit tests do not raise any warnings with this patch.

Can it be merged?

I'll bump this one last time.

Here is a new patch addressing comments raised in review. It supersedes previous patch submissions.

Patch addressing latest comments in review. Notable change: defines _PyObject_IsAbstract in object.c/object.h, rather than repeating the code in multiple files and functions.

Added some review comments in Reitveld - core functionality changes look good, but there are some other aspects that need addressing before the patch will be good to go (primarily relating to only doing a minimal documented deprecation of the legacy APIs without actually harming their documentation, testing or functionality).

New patch addressing comments in review.

Is this patch ready to go? I haven't heard any feedback on the most recent version.

New changeset e979b26a9172 by Benjamin Peterson in branch 'default':
improve abstract property support (closes #11610)
http://hg.python.org/cpython/rev/e979b26a9172