Super() walks the MRO and looks for the looked-up attribute in the __dict__ of types on the MRO. This can cause problems when classes implement __getattribute__: a class on the MRO can implement a method that super() won't find because it isn't in the class __dict__ (yet).

I'm running into this with PyObjC: the __dict__ for the Python proxies for Objective-C classes are filled lazily (*) by tp_getattro (__getattribute__ in Python) to speed up the bridge and because ObjC is almost as dynamic as Python and methods might appear during runtime (without there being a hook for detecting such changes).

A possible solution to this:

* Add a tp_getattro_super (**) slot to PyTypeObject, with the 
  same signature as tp_getattro, but that only looks at this particular
  class (as opposed to tp_getattro that walks the entire MRO and looks
  in the object's __dict__)  (***)

* The tp_gettro of super calls tp_getattro_super of types of along the
  MRO when that slot is not NULL, and uses the current implementation
  (look in tp_dict) when the slot is NULL.

Would such a change be acceptable?

Open issues:

* Does the new slot get exposed to Python code 
  (and if so, under which name)?

* Should PyObject_GenericGetAttr use the new slot as well?


Footnotes:

(*) The current release of PyObjC (2.5) eagerly tries to keep the proxy class __dict__ up to date, an upcoming major release will be as lazy as possible to speed up the bridge. The problem can with some effert be triggered with PyObjC 2.5, and triggering it is easy in the upcoming major release

(**) Or some better name

(***) I'm being very sloppy in my use of terminology here, hopefully my proposal is clear enough anyway.

Do you have an example in pure Python code?

I didn't, but the attached script should do the trick. The code suffers badly from copy&paste editing, but more or less has the same behavior as PyObjC.

It defines a subclass of list (MyList) with an append method, and more importantly also defines 3 "proxy" classes: ProxyObject, ProxyList and ProxyMyList that are proxy types for object, list and MyList.

At the end I try to access methods on instances of a proxy objects for MyList, and finally try to access super(MyList, v).append. That only works when "append" is in the __dict__ for ProxyList (for example by uncommenting the 3 lines just above the use of super()).

issue783528 was a bit similar.

Looks like it. That issue was closed without a good reason though.

I've attached a first very rough patch to demonstrate what I was rambling about.

With this patch and a patched version of PyObjC I can use the builtin super class with the proxy classes for Objective-C classes.

Open issues:

* Does this need a pep?

* The name of the new slot sucks, need to find a better one

* Should it be possible to define the slot in Python code?

  Probably, the 'super.py' file is a crude hack but I can imagine
  more realistic scenarios where implementing the slot in Python
  would be useful.

* There are no tests, and no documentation updates

Second version of the poc adds a way to implement the hook in Python code, and supers_updated.py uses that hook.

Still no tests, no documentation and with awfull naming :-)

.

Is it possible to avoid a new slot?
for example, super() could walk the tp_base-> chain and call tp_getattro each time the value changes.

I don't think it is possible to avoid adding a new slot. The default implementation of tp_getattro (PyObject_GenericGetAttr) looks in the instance dict, while super does not (because it wants a less specific implementation).  

PyObject_GenericGetAttr will also walk the entire MRO, while super.__getattribute__ does not start search at the first entry in the MRO.

Although I'd love to be proven wrong ;-)

Slightly updated patch (different names, added some documentation)

added draft of a pep-style proposal

Updated version of the patch, with some semantic changes and tests for the Python level API.

Known issues:

* Patches uses tabs for indentation

* No tests for the C level API

* I don't like implementation of slot_tp_getattro_super

Also updated the draft PEP text

There now is a PEP for this issue: <http://www.python.org/dev/peps/pep-0447/>.

The current version of the PEP describes a different API than documented by the attached proposal and the prove of concept code. I'm working on an updated implementation.

issue18181-locallookup-only-super.txt implements the current version of PEP 447, but only for the super object. In this version of the  patch normal attribute lookup (object.__getattribute__ and PyObject_GenericGetAttr) are not yet affected.

Another open issue: the documentation for the super object itself does not yet mention __locallookup__; it probably should but I don't know yet how to cleanly add it there.

issue-18181-full-v1.txt implements support for __locallookup__ to both super and _PyType_Lookup and should implement the entire PEP.

The patch is not yet 100% and is missing:

  * Tests that add __locallookup__ to a metaclass at runtime, should
    enable the new behavior

  * Tests that use the tp_locallookup slot in an type defined in C code

  * Documentation has not been updated.

 
Changes to the implementation:

  * _PyType_Lookup is now _PyType_LookupName 
    and returns a new reference (not a borrowed one)

  * _PyType_Lookup and super.__getattribute__ use the 
     __locallookup__ method when present

  * __locallookup__ is not defined on "type", 
    and the type attribute lookup cache is disabled
    for types that have a metatype with __locallookup__

  * The type attribute cache is disabled for types that are 
    not ready (Py_TPFLAGS_READY), primarily
    as a side effect of the implementation.

    The cache is also disabled for types with a metaclass that
    has a tp_locallookup attribute, as mentioned in the PEP.

    The code does not change the VERSION flags of the type because
    dynamicly adding a __locallookup__ method to the metaclass would
    have to change all instances of that metaclass and that's not
    easily possible.

issue-18181-full-v2.txt adds more tests to v1 and has slightly better documentation (still not good enough).

A major issue with the patch: for some reason the tests pass when run standalone ("./python.exe -m test.regrtest test_pep447"), but crash when I run another test before it (such as when running "make test").

I'm debugging the problem (this is probably a refcounting issue, but you never know...)

The crash was due to a bug in the test extension. Fixed in version 2b.

* Improved docmentation

* __locallookup__ is now called unconditionally, the attribute lookup
  cache is now disabled when aType.__locallookup__ != type.__locallookup__

  This gives a slightly nicer API (the only reason the new method
  was optional was due to an implementation detail in _PyType_Lookup)

Updated the patch for PEP 442.

Updated the title and versions, will work on an updated PEP and patch during and after EP'15.

I've attached a new version of the patch (pep447-2015-07-25.txt). Changes in this version of the patch:

1) Works with the current trunk (as in "all tests pass")

2) Types in C must explicitly set Py_TPFLAGS_GETDESCRIPTOR in tp_flags to 
   enable the tp_getdescriptor slot. 

   This is primarily done avoid crashing when loading a C extension for
   older CPython versions (even if that's not guaranteed to work anyway).

The PEP needs to be updated for the second change, that's next on my list.

The attached micro benchmark indicates that method_cache in typeobject.c isn't used when using my patch. I'll have too look into that.

Other than that benchmarks results look OK (but: not using the method_cache is unacceptable as this most definitely changes performance).

What is kind of scary, but is to be expected I guess, is that a __getdescriptor__ method in Python is awfully slow.  I have to look into why this is slow, as the slowdown for using Python is significantly worse than I expected.

pep447-2015-07-25-v2.txt changes two things w.r.t. the patch earlier today:

1) The performance problems w.r.t. the method_cache are gone

2) Added code for handling python exceptions other than AttributeError in
   calls to __getdescriptor__. 

   This code requires some tests to make sure the error handling code 
   is correct.

The new patch also adds some more tests, and adds extra MCACHE_STATS counters. Those aren't part of the proposal but were helpful to find the problem w.r.t. 1).

As I mentioned before implementing __getdescriptor__ in Python is bad for performance of instances of that metaclass. AFAIK that cannot be helped, doing that will just execute a lot more Python code.

Note: the error handling code for exceptions in __getdescriptor__ definitely isn't good enough yet. 

I'm writing tests and am hunting down the problems those tests find. I'm getting closer and will post a new version when I think I've found all bugs. Probably some time tomorrow.

For now be warned that unexpected exceptions in __getdescriptor__ might crash the interpreter, especially when it is used to look for dunder methods.

Todays version of the patch adds more tests, especially for exceptions in __getdescriptor__ during the lookup of special methods.  Those tests were added because the C code couldn't get exceptions other than AttributeError before this patch and can get those know. This requires extra code, and hence extra tests (which exposed a number of crashers...).

The code should be solid now.

Note that the new error handling code isn't optimal style-wise, at some places I test for PyErr_Occurred() instead of adding error returns to functions now returning 'void' to keep the patch smaller and easier to review.