in urllib2, you will find these lines:
        # Wrap the HTTPResponse object in socket's file object adapter
        # for Windows.  That adapter calls recv(), so delegate recv()
        # to read().  This weird wrapping allows the returned object to
        # have readline() and readlines() methods.

        # XXX It might be better to extract the read buffering code
        # out of socket._fileobject() and into a base class.

        r.recv = r.read
        fp = socket._fileobject(r, close=True)

This, storing a bound method in the instance, will cause a reference 
cycle that the user knows nothing about.

I propose creating a wrapper instance with a recv() method instead.  Or, 
is there a standard way of storing bound methods on instances?  A 
'weakmethod', perhaps?

I have two solutions for this problem.  The first is a mundane one, and 
what I employed in our production environment:
class RecvAdapter(object):
    def __init__(self, wrapped):
        self.wrapped = wrapped
    def recv(self, amt):
        return self.wrapped.read(amt)
    def close(self):
        self.wrapped.close()

...
    fp = socket._fileobject(RecvAdapter(r), close=True)


The second solution is a bit more interesting.  It involves applying 
what I call a weakmethod: A bound method that holds a weak ref to the 
object instance:

import weakref
class WeakMethod(object):
    def __init__(self, bound):
        self.weakself = weakref.proxy(bound.im_self)
        self.methodname = bound.im_func.func_name
    def __call__(self, *args, **kw):
        return getattr(self.weakself, self.methodname)(*args, **kw)

We then do:
    r.recv = WeakMethod(r.read)
    fp = socket._fileobject(r, close=True)

I've had many uses for a WeakMethod through the years.  I wonder if such 
a class might be considered useful enough to be put into the weakref 
module.

The WeakMethod idea is not new:

http://code.activestate.com/recipes/81253/

http://mindtrove.info/articles/python-weak-references/#toc-extending-weak-
references

weak method idea seems interesting. I have not used it anytime yet, and
in this case,it seems okay.

cool.  The mindtrove one in particular seems nice.  I didn't realize 
that one could build boundobjects oneself.
Is there any chance of getting a weakmethod class into the weakref 
module?

On Fri, Dec 11, 2009 at 09:40:40AM +0000, Kristján Valur Jónsson wrote:
> Is there any chance of getting a weakmethod class into the weakref 
> module?

This is a separate feature request on weakref module. It may opened
and discussion carried out there?

Lets just investigate the circular reference part here for this ticket.

Does this issue still exist? I did a little poking around at could not find the quoted code.

in python/trunk/Lib/urllib2.py, line 1161

It doesn't appear to be an issue in py3k.

This is still a horrible, horrible, cludge.
I've recently done some work in this area and will suggest a different approach.

Could you please provide some tests?

Here it is.
Notice the incredible nesting depth in python 2.7.
The socket itself is found at
response.fp._sock.fp._sock
There are two socket._fileobjects in use!

Sounds like urlopen() is relying on garbage collection to close the socket and connection. Maybe it would be better to explicitly close the socket, even if you do eliminate all the garbage reference cycles.

My test code for Issue 19524 might be useful here. It verifies close() has been called on the HTTP socket.

No, the socket is actually closed when response's close() method is called.  The problem is that the HTTPResponse object, buried deep within the nested classes returned from do_open(), has a circular reference, and _it_ will not go away.

No one is _relying_ on garbage collection in the sense that this is not, I think, designed behaviour, merely an unintentional effect of storing a bound method in the object inance.

As always, circular reference should be avoided when possible since relying on gc is not something to be done lightly.

Now, I think that changing the complicated wrapping at this stage is not possible, but merely replacing the bound method with a weak method might just do the trick.

> It doesn't appear to be an issue in py3k.

I agree, I can't find this code anywhere. Closing.