Author tim.peters
Recipients
Date 2006-09-26.10:04:58
SpamBayes Score
Marked as misclassified
Message-id
In-reply-to
Content
Logged In: YES 
user_id=31435

Your memory bloat is mostly due to the

d = range(100000)

line.  Python has no problem collecting the cyclic trash,
but you're creating 100000 * 100 = 10 million integer
objects hanging off trash cycles before invoking
gc.collect(), and those integers require at least 10 million
* 12 ~= 120MB all by themselves.  Worse, memory allocated to
"short" integers is both immortal and unbounded:  it can be
reused for /other/ integer objects, but it never goes away.

Note that memory usage in your program remains low and
steady if you force gc.collect() after every call to bar().
 Then you only create 100K integers, instead of 10M, before
the trash gets cleaned up.

There is no simple-minded way to "repair" this, BTW.  For
example, /of course/ a frame has to reference all its
locals, and moving to weak references for those instead
would be insanely inefficient (among other, and deeper,
problems).

Note that the library reference manual warns against storing
the result of exc_info() in a local variable (which you're
/effectively/ doing, since the formal parameter `s` is a
local variable within foo()), and suggests other approaches.
 Sorry, but I really couldn't tell from your description why
you want to store this stuff in an instance attribute, so
can't guess whether another more-or-less obvious approach
would help.

For example, no cyclic trash is created if you add this
method to your class O:

    def get_traceback(self):
        self.e = sys.exc_info()

and inside foo() invoke:

    s.get_traceback()

instead of doing:

    s.e = sys.exc_info()

Is that unreasonable?  Perhaps simpler is to define a
function like:

def get_exc_info():
    return sys.exc_info()

and inside foo() do:

    s.e = get_exc_info()

No cyclic trash gets created that way either.  These are the
kinds of things the manual has suggested doing for the last
10 years ;-)
History
Date User Action Args
2007-08-23 14:43:01adminlinkissue1565525 messages
2007-08-23 14:43:01admincreate