This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Weakproxy is an instance of collections.Iterator
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.4, Python 3.5, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, ereuveni, fdrake, gdr@garethrees.org, gvanrossum, pitrou
Priority: normal Keywords:

Created on 2015-04-27 21:42 by ereuveni, last changed 2022-04-11 14:58 by admin.

Messages (13)
msg242159 - (view) Author: Eyal Reuveni (ereuveni) Date: 2015-04-27 21:42
Calling weakref.proxy() on an object returns something that is an instance of collections.Iterator, regardless of whether the object is an instance of collections.Iterator or even if the object itself is iterable.

To reproduce, run the following code (verified in python2.7 and 3.4 on my Mac):

import collections
import weakref

class NotIterator:
    pass

not_iterator = NotIterator()
isinstance(not_iterator, collections.Iterator) # returns False

proxy_object = weakref.proxy(not_iterator)
isinstance(proxy_object, collections.Iterator) # returns True, should be False
msg242172 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2015-04-28 10:00
Not just Iterator, but Container, Hashable, Iterable, and Sized too!

    >>> import weakref
    >>> class C: pass
    >>> o = C()
    >>> w = weakref.proxy(o)
    >>> from collections.abc import *
    >>> isinstance(w, Container)
    True
    >>> isinstance(w, Hashable)
    True
    >>> isinstance(w, Iterable)
    True
    >>> isinstance(w, Sized)
    True
msg242174 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 10:14
Those methods are defined precisely so that they can be delegated. Since virtually anything can be proxied, the weakref.proxy *class* cannot tell upfront whether the proxied object will be an Iterator or not. I don't really see how to get around that.
msg242175 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2015-04-28 11:08
Hashable is particularly misleading, because "weakref.Proxy objects are not hashable regardless of the referent".
msg242177 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 11:38
> weakref.Proxy objects are not hashable regardless of the referent

Interesting. @fdrake, do you remember why that is?
msg242178 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2015-04-28 11:42
The documentation says that weakref.Proxy objects are not hashable because "this avoids a number of problems related to their fundamentally mutable nature, and prevent their use as dictionary keys".

Hashable objects must be immutable, otherwise the hash might change, invalidating the invariants that make dictionaries work, but Proxy objects are fundamentally mutable: when there are no more strong references to the proxied object, the object gets destroyed and the Proxy object now refers to None. If the Proxy object were hashable then its hash would change at this point.
msg242179 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 11:50
Still, weakref.ref objects are hashable, so I don't understand why weakref.proxy would be different.

Hashability of weakref.ref has been added in 29aa832b8787, to support weak dictionaries. See issue403985. It simply wasn't discussed whether to also add hashability to proxy objects.
msg242180 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 11:54
Ok, PEP 205 explains it a bit more:

"the resulting proxy cannot be used as a dictionary key since it cannot be compared once the referent has expired, and comparability is necessary for dictionary keys. Operations on proxy objects after the referent dies cause weakref.ReferenceError to be raised in most cases."

Perhaps this can be relaxed, and comparison simply made to return false. The following behaviour is a bit troubling :-)

>>> p
<weakproxy at 0x7f4054fe8cd8 to NoneType at 0x88a780>
>>> p == p
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ReferenceError: weakly-referenced object no longer exists
msg242181 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2015-04-28 12:37
I don't see any reason for proxy objects to be less hashable than ref objects.

As for the p == p case, where the referent has expired, returning True if p is p seems acceptable (along with False inequalities, and True for other comparisons allowing equality), but anything beyond that seems unwise.  Not sure whether that would really be enough to help real use cases.
msg242182 - (view) Author: Gareth Rees (gdr@garethrees.org) * (Python triager) Date: 2015-04-28 12:54
> I don't see any reason for proxy objects to be less hashable than ref objects.

The difference is that unlike a ref object, a proxy object is supposed to forward its method calls to the proxied object. So consider what happens if you forward the __hash__ method to the proxied object: the hash will change when the object dies.

A proxy object could, of course, not forward the __hash__ method, instead computing its own hash. But I think this would do more harm than good: surely most attempts to store weakref.Proxy objects in sets or dictionaries are going to be mistakes -- the user should have used a WeakKeyDictionary or a WeakSet instead.
msg242184 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2015-04-28 14:06
Clearly I've been away from this code for a long time.

The hash support for ref objects is definitely a very special case, only intended to support WeakKeyDictionary.  We that class implemented in C, we'd probably want the hash support for refs not to be exposed.

You've convinced me hashability for proxies isn't desirable.  Let's stick with the status quo on this.
msg242186 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-28 14:36
> So consider what happens if you forward the __hash__ method to the
> proxied object: the hash will change when the object dies.

ref objects behave differently: they inherit their referent's hash value when alive, and remember it.  proxy objects could be made to behave the same way.

> The hash support for ref objects is definitely a very special case, 
> only intended to support WeakKeyDictionary

I've relied several times on the hashability of ref objects, in third-party code.
(OTOH, I never use weakref proxies)
msg242187 - (view) Author: Fred Drake (fdrake) (Python committer) Date: 2015-04-28 14:50
> ref objects behave differently: they inherit their referent's hash
> value when alive, and remember it.  proxy objects could be made to
> behave the same way.

They could, yes, but that would break the proxy behavior, and the hash <--> equality behavior for mutable objects.

In particular, mutable objects can become equal; if the hashes were computed for the proxies before that happened, the hashes would be inappropriate later.  That's pretty important.
History
Date User Action Args
2022-04-11 14:58:16adminsetgithub: 68255
2015-04-28 14:50:36fdrakesetmessages: + msg242187
2015-04-28 14:36:39pitrousetmessages: + msg242186
2015-04-28 14:06:26fdrakesetmessages: + msg242184
2015-04-28 12:54:51gdr@garethrees.orgsetmessages: + msg242182
2015-04-28 12:37:16fdrakesetmessages: + msg242181
2015-04-28 11:54:38pitrousetmessages: + msg242180
2015-04-28 11:50:31pitrousetmessages: + msg242179
2015-04-28 11:42:46gdr@garethrees.orgsetmessages: + msg242178
2015-04-28 11:38:41pitrousetmessages: + msg242177
2015-04-28 11:08:30gdr@garethrees.orgsetmessages: + msg242175
2015-04-28 10:14:21pitrousetnosy: + gvanrossum, benjamin.peterson

messages: + msg242174
versions: + Python 3.5
2015-04-28 10:00:53gdr@garethrees.orgsetnosy: + gdr@garethrees.org
messages: + msg242172
2015-04-28 01:54:51ned.deilysetnosy: + fdrake, pitrou
2015-04-27 21:42:46ereuvenicreate