The lookup of special methods invoked implicitly by
syntax other than explicit instance.attribute changed
to *not* use __gettattr__ when normal lookup failed.
This is contrary to docs, which consistently say
__getattr__ is unchanged. New special method
__getattribute__ is also bypassed, contrary to
implication of doc-essay.

This was reported on c.l.p by Jan Decaluwe using
2.2.2 on Red Hat Linux. On Win98, 2.2.1, I get same
behavior (I added test of __getattribute__):

class Iold:
  def __init__(self,ob):
    self.ob = ob
  def __getattr__(self,name):
    return getattr(self.ob,name)

class Inew(object):
  def __init__(self,ob):
    self.ob = ob
  def __getattr__(self,name):
    return getattr(self.ob,name)

a = Iold(1) #2
b = Inew(1) #2
a.__add__(1) #2
b.__add__(1) #2
a+1 #2
b+1 #error
#TypeError: unsupported operand types for +: 'Inew' 
and 'int'
Inew.__getattribute__ = Inew.__getattr__
b+1 #same error, no loop
#TypeError: unsupported operand types for +: 'Inew' 
and 'int'

b.__add__(1) # WARNING starts 'infinite' loop

def _(self,other): print 'hello'

Inew.__add__ = _
b+1 #prints 'hello', __getattribute__ bypassed.

http://python.org/2.2/descrintro.html says:
"Note that while in general operator overloading
works just as for classic classes, there are some
differences. (The biggest one is the lack of support for
__coerce__; new-style classes should always use the
new-style numeric API, which passes the other
operand uncoerced to the __add__ and __radd__
methods, etc.) "

Was lookup change meant to be one of differences?

"There's a new way of overriding attribute access. The
__getattr__ hook, if defined, works the same way as it
does for classic classes: it is only called if the regular
way of searching for the attribute doesn't find it."

But it is different.

"But you can now also override __getattribute__, a
new operation that is called for all attribute
references."

Except for implicit special methods.

I did not classify discrepancy because I don't know
whether interpreter or docs are off.

Logged In: YES
user_id=593130

Print-style 'debugging' output provided by Bengt Richter in
a follow-up in the c.l.p. thread "Issue with new-style
classes and operators" showed that 'a+1' worked because
of a __getattr__ call getting '__coerce__' (rather
than '__add__') and that 'b+1' did not trigger such a call.
So I presume the quoted parenthesized statement about
__coerce__ and 'new-style numeric API' was meant to
explain as least this part of the change in behavior.
However, it is still not clear to me, even after reading the
development (2.3a) version of the ref manual, why failure
to find '__add__' in 'the usual places' (to quote RefMan
3.3.2 __getattr__ entry) does not trigger a call to
__getattr__, nor why the initial attempt to find it did not
trigger __getattribute__. The sentence 'For objects x and y,
first x.__op__(y) is tried' (3.3.7) implies to me that there is
an attempt to find __op__ as an attribute of x. Unless I
missed something I should have found, a bit more
clarification might be usefully added.

Logged In: YES
user_id=1038590

FWIW, this is still true for Python 2.4 (i.e. the search for
'__add__' by '+' invokes neither __getattr__ nor
__getattribute__).

The 'infinite loop' is courtesy of the call to 'self.ob'
inside __getattr__.

Also of interest - int(b) fails with a TypeError, and str(b)
returns the generic object representation of b, rather than
delegating to the contained object.

So it looks like these methods are not invoked when looking
up any 'magic methods' recognised by the interpreter.
Inspection of PyNumber_Add shows that this is the case - the
objects' method pointers are inspected directly at the C level.

To me, this looks like a documentation bug. Magic methods
are found by checking if the associated slot in the method
structure is not NULL, rather than by actually looking for
the relevant magic 'attribute'.

In order to change the 'implicit' behaviour of the object,
it is necessary to change the contents of the underlying
method slot - a task which is carried out by type whenever
an attribute is set, or when a new instance of type is created.

I was just bit by this today in converting a proxy class from old style
to new style. The official documentation was of no help in discoverting
that neither __getattr__ or __getattribute__ are used to look up magic
attribute names. Even the link to "New-style Classes" off the
development documentation page is useless, as none of the resources
there (http://www.python.org/doc/newstyle/) mention the incompatible change.

This seems like an issue that is going to come up more frequently as
python 3000 pushes everyone to using only new style classes. It'd be
very useful if whatever conversion tool we get, or the python 3000
standard library includes a proxy class or metaclass that is able to
help with this conversion, such as this one:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/252151

Though preferably with some knowledge of all exising magic names.

The 2.6 and 3.0 documentation has already been updated appropriately.
The forward compatible way to handle this is to inherit from object and
override the special methods explicitly in the 2.x version (Yes this can
make writing proxy objects more tedious, but from our experience with
the tempfile module, I would say that the bulk of proxy objects that
aren't overriding special methods on a case-by-case basis are probably
broken anyway).

It may be appropriate to add a -3 warning that is triggered whenever a
special method is retrieved via __getattr__ on a classic class.

I assume when you say that the documentation has already been updated,
you mean something other than what's shown at:
http://docs.python.org/dev/reference/datamodel.html#new-style-and-
classic-classes
or
http://docs.python.org/dev/3.0/reference/datamodel.html#new-style-and-
classic-classes ?

As both of those claim to still not be up to date in relation to new
style classes, and the __getattr__ & __getattribute__ sections under
special names make no reference to how magic methods are handled.
Additionally, the "Class Instances" section under the type heirachy
makes mention of how attributes are looked up, but doesn't mention the
new style differences even in the 3.0 documentation.

Sure, there's this note under "Special Method Names":
For new-style classes, special methods are only guaranteed to work if
defined in an object’s class, not in the object’s instance dictionary.

But that only helps you figure it out if you already know what the
problem is, and it's hardly comprehensive.

I'm not arguing that this is something that's going to change, as we're
way past the point of discussion on the implementation, but this looks
far more annoying if you're looking at it from the perspective of
proxying to container classes or numeric types in a generic fashion. My
two use cases I've had to convert are for lazy initialization of an
object and for building an object that uses RPC to proxy access to an
object to a remote server.

In both cases, since they are generic proxies that once initialized are
supposed to behave exactly like the proxied instance, the list of magic
methods to pass along is ridiculously long. Sure, I have to handle
__copy__ & __deepcopy__, and __getstate__ & __setstate__ to make sure
that they return instances of the proxy rather than the proxied object,
but other than that there's over 50 attributes to override for new
style classes just to handle proxying to numeric and container types.

It's hard to get fancy about it too, as I can't just dynamically add
the needed attributes to my instances by looking at the object to be
proxied, it really has to be a static list of everything that python
supports on the class. Additionally, while metaclasses might help here,
there's still the problem that while my old style proxy class has
continued to work fine as magic attributes have been added over python
revisions, my new style equivalent will have to be updated work
currectly as magic methods are added. Which, given the 2.x track seems
to happen pretty frequently. Some of the bugs from that would have been
quite tricky to track down too, such as the __cmp__ augmentation with
the rich comparison methods.

None of the solutions really seem ideal, or at least as good as what
old style classes provided, which is why I was hoping for some support
in the 3.0 standard library or the conversion tool.

Agreed that section of the docs should be more explicit in pointing out
that __getattr__ and __getattribute__ won't work for proxying special
methods for new-style classes.

It's also true that there are quite a few special methods where nothing
special is needed to correctly proxy the methods. That said, defining
all of the methods blindly is also bad (as a lot of informal type
testing is done by checking for certain special methods as attributes).

Perhaps it would be useful to have a method on type instances that could
be explicitly invoked to tell the type to run through all the tp_* slots
and populate them based on doing getattr(self, "__slotname__").

(Note that it's only those methods with dedicated slots in the C type
structure that are a problem here - those which are implemented solely
as normal Python methods like __enter__, __exit__ or the pickle module's
special methods can typically be overridden as expected through the use
of __getattr__ and __getattribute__)

I've started a discussion on the Py3k development list regarding the
possibility of adding a new method to type to aid in converting proxy
classes from classic- to new-style.

I spent an enlightening evening browsing through the source code for
weakref.proxy. The way that code works is to define every slot,
delegating to the proxied object to handle each call (wrapping and
unwrapping the proxied object as needed).

This is normally transparent to the user due to the fact that
__getattribute__ is one of the proxied methods (and at the C level, the
delegated slot invocations return NotImplemented or set the appropriate
exceptions). The only way it shows through is the fact that
operator.isNumber and operator.isMapping will always return True for the
proxy instance, and operator.isSequence will always return False - this
is due to the proxy type filling in the number and mapping slots, but
not the sequence slots.

However, this prompted me to try an experiment (Python 2.5.1), and the
results didn't fill me with confidence regarding the approach of
expecting 3rd party developers to explicitly delegate all of the special
methods:

>>> class Demo:
...   def __index__(self):
...     return 1
...
>>> a = Demo()
>>> b = weakref.proxy(a)
>>> operator.index(a)
1
>>> operator.index(b)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'weakproxy' object cannot be interpreted as an index

Oops.

Somewhat related: bpo-2605 (descriptor __get__/set/delete)

I've been following the py3k maliing list disscussion for this issue,
and wanted to add a note about the proposed solution described here:
http://mail.python.org/pipermail/python-3000/2008-April/013004.html

The reason I think this approach is valuable is that in all of the
proxy classes I've written, I'm concerned about which behaviour of the
proxied class I want to override, not which behaviour I want to keep.
In other words, when I proxy something, my mental model has always
been, okay, I want something that behaves just like X, except it does
this (usually small bit) differently.

This is also why I expect my proxies to keep working the same when I
change the proxied class, without having to go and update the proxy to
also use the new behaviour.

So, yeah, very much in favor of a base proxy class in the standard
library.

I've attached a sample ProxyBase class that delegates all of the special
methods handled by weakref.proxy (as well as the tp_oct and tp_hex
slots, which weakref.proxy ignores).

While there are other special methods in CPython (e.g. __enter__ and
__exit__), those don't bypass the normal lookup mechanism, so the
__getattribute__ override should handle them.

For lack of a better name, I called the module typetools (by analogy to
functools).

Note that correctly implementing a proxy class as a new-style class
definitely turns out to be a fairly non-trivial exercise (and since this
one is still sans-tests, I don't make any promises that even it is 100%
correct at this point)

Is there any reason not to name it ProxyMixin, ala DictMixin?

Attached a new version of the module, along with a unit test file. The
unit tests caught a bug in the __gt__ implementation. I've also changed
the name to ProxyMixin as suggested by Adam and switched to using a
normal __init__ method (there was no real gain from using __new__ instead).

Added documentation, and assigned to Barry as release manager for 2.6/3.0.

Also bumped to 'release blocker' status because I think the loss of
classic classes transparent proxying capabilities is a fairly
substantial issue that needs to be addressed explicitly before the first
3.0 beta.

If I get the go-ahead from Barry or Guido, I'll add the new module to
2.6 (from whence it will be migrated to 3.0 as part of the normal merge
process).

Also changed to a library issue instead of a docs issue.

Unfortunately, the standard library doesn't tend to do this kind of
delegation (aside from weakref.proxy, which implements the equivalent
code directly in C), so there isn't a lot of standard library code that
will benefit from it directly.

However, maintaining such a class on PyPI is also fairly undesirable,
due to the extremely tight coupling between the list of methods it needs
to delegate explicitly and the tp_* slots in the PyType method
definitions - add a new tp_* slot, and it's necessary to add the same
methods to the proxy class. Different Python implementations are going
to have different needs as to which slots they have to delegate
explicitly, and which can be left to the __getattribute__ catch-all
delegation.

As far as adding a module for a single class goes, I wouldn't expect it
to remain that way forever. E.g., I'd hope to eventually see a
CallableMixin that defined __get__ the same way a function does, making
it easier to create callables that behave like functions when stored as
a class attribute.

That said, I'd be happy enough with adding the ProxyMixin to the types
module instead, but I thought we were still trying to get rid of that
module.

And (mainly for Barry's benefit) a quick recap of why I think this is
necessary for Python 3.0:

For performance or correctness reasons, the interpreter is permitted to
bypass the normal __getattribute__ when looking up special methods such
as __print__ or __index__. Whether or not the normal attribute lookup
machinery is bypassed for a specific special method is an application
independent decision.

In CPython's case, this bypassing can occur either because there is a
tp_* slot dedicated to the method, or because the interpreter uses
Py_TYPE(obj) and _PyType_Lookup instead of PyObject_GetAttr to find the
method implementation (or type(obj).meth instead of obj.meth for special
method lookups implemented in Python code).

This behaviour creates a problem for value-based delegation such as that
provided by weakref.proxy: unlike overriding __getattr__ on a classic
class, merely overriding __getattribute__ on a new-style class instance
is insufficient to be able to correctly delegate all of the special methods.

The intent of providing a typetools.ProxyMixin (or alternatively a
types.ProxyMixin class) is to allow fairly simply conversion of classic
classes that implement value-based delegation to new-style classes by
inheriting from ProxyMixin rather than inheriting from object directly.

Given the close proximity of the beta perhaps I should PEP'ify this to
get a formal yea or nay from Guido? I haven't managed to get much
response to previous python-dev posts about it.

bleh, "application independent decision" in my last post should read
"interpreter implementation dependent decision".

and "__print__" was meant to be "__unicode__"...

New patch (proxymixin.diff) uploaded that correctly delegates
__format__, as well as using an overridable return_inplace() method to
generate the inplace operation return values. The _target attribute has
also been made formally part of the public API (as 'target'), although
you obviously need to explicitly invoke object.__getattribute__ in order
for it to be visible. The name of the attribute is also available at the
module level as _PROXY_TARGET_ATTR.

Note that I don't make any promises about the correctness of the ReST
formatting in that latest patch - my Doc build is misbehaving at the
moment, and I haven't had a chance to look at what is wrong with it.

The name Proxy seems too vague. This class is all about targeted
delegation. Am curious, has this been out as a recipe; has it been
used in combat yet?

I want to make this "bypass getattr" behavior mandatory for those
operations that currently use it, forcing the issue for other
implementations of Python. That's a doc change (but an important one!).
There are probably many random places where the docs imply that getattr
is used where it isn't.

I am not sure that we need a proxy implementation in the stdlib; usually
when proxying there is some intentional irregularity (that's why you're
proxying) and I'm not sure how useful the mix-in class will be in
practice. We should wait and see how effective it is in some realistic
situations before accepting it into the stdlib. Also, typetools strikes
me as a horrible name.

Thanks for the pronouncement Guido. We will not let this issue hold up
the beta releases.

The outcome of discussion of this issue on python-dev was that the
lookup methodology for the special methods needs to be better
documented, especially for those cases where the instance *must* be
bypassed in order to avoid metaclass confusion for special methods that
apply to both types and instances (see bpo-2517 for such a problem
that currently afffects the lookup of __unicode__). However, we're not
prepared to add a standard delegation mixin to the standard library at
this stage (I may still add a cleaned up version of mine to the SVN
sandbox as an executable reference source for the relevant section of
the documentation though).

While I offered to write that new section of the docs during the
python-dev discussion, I'm not sure when I'll be able to get to it (My
Python time lately has mostly been spent investigating __hash__ fun and
games).

Hi, I was trying to implement a generic proxy class with some
restricting behaviors and then I bumped into this complication. May I
ask what the conclusion is going to be in the long run?

I guess my question is that

1> is it just going to be a documentation change (first)?

2> is it technically difficult(other things are going to be broken
because of this change) or very inefficient to make the behavior of
special method lookup same as normal methods?

2> Otherwise, what is the rationale behind keeping the differences?

thank you.

There are both speed and correctness reasons for special methods being
looked up the way they are, so the behaviour isn't going to change.

Aside from providing additional details in the language reference on how
special methods are looked up (and the reasons things are done that
way), the only change that might eventually happen in this area is the
inclusion of a mixin class in the standard library to assist in writing
classes which delegate most operations to another object. That part
won't happen before 2.7/3.1 though (if it happens at all).

Attaching a documentation patch for the moment until I get some info
back from Georg as to why I can't build the docs locally.

Once I get my local doc build working again, I'll check the formatting
and check it in.

Committed for 2.6 as r65487.

I also blocked the automatic merge to 3.0 since the references to
old-style classes don't make sense there.

But don't the docs with patch describe the behavior of new-style classes
better?

I meant to say that I will be merging it manually to avoid bringing the
old-style class specific parts over (that's why I left the issue open
and assigned to me).

Ah, I'm sorry for the noise then.

Docs updated for 3.0 in r66084 (and I was right in thinking the
automatic merge didn't have a hope of getting this right - there were
conflicts all the way through the file).

Closing this issue after a mere 5 years and 9 months - any requests for
better proxying/special method delegation support in the standard
library should be raised in a new tracker issue as a feature request for
2.7/3.1.