This patch returns more memory to the system when doing:

>>> x = [unicode(i) for i in xrange(1000000)]
>>> del x

If the above code is done, the memory before and after is quite
different. After this patch, the memory of the process as reported by
the system (like top/ps) should be approximately the same

Hi Neal,

This seems to be a more general problem than just unicode.

eg. Tuples:

>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]
>> del x

And user-defined objects:

>>> class A(object):
...   def __init__(self):
...     self.x = random.random()
>>> x = [A() for i in xrange(800000)]
>>> del x

Both exhibit the same behaviour. Naively to me it seems like using the
custom allocator uniformly would fix this problem.

On Mon, Mar 17, 2008 at 11:55 AM, Alec Thomas <report@bugs.python.org> wrote:

Alec Thomas <alec@swapoff.org> added the comment:

Hi Neal,

This seems to be a more general problem than just unicode.

Kinda, sorta. The general issue is the pattern of memory
allocation/deallocation. In the case of

>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]

The memory that is not returned is in the integer free list. If this
code is changed to:

>> for x in ((1, 2, 3, 4, i) for i in xrange(800000)): pass

That doesn't hold on to any extra memory. The problem is that holes
are left in memory and a lot of it can't always be returned to the
O/S. It can still be re-used by python.

Both exhibit the same behaviour. Naively to me it seems like using the
custom allocator uniformly would fix this problem.

In general, we should find places (like unicode) that use PyMem_* (ie,
system malloc) and replace them with PyObject_* (pymalloc). That
should improve the behaviour, but there will always be some allocation
patterns that will be suboptimal. Note that using pymalloc will only
help for objects < 256 bytes. Larger objects are delegated to the
system malloc and will still exhibit some of the bad problems.

Alec, can you find places that are using the PyMem_* interface and
create a patch to fix them. That would be great!

Alec, can you find places that are using the PyMem_* interface and
create a patch to fix them. That would be great!

Sure thing! I'll see if I can finish it today, but if not I'll work on
it during the flight to Zurich tonight.

Just for the record, the enhanced profiler source files in
<http://bugs.python.org/issue2281\> and <http://bugs.python.org/issue2218\>
do replace calls to malloc and free with PyObject_MALLOC resp. _FREE.

Looks good, Neal. Are you hesitant to check this in? I ran a little
test showing that indeed it gives much more memory back to the system.

After discussing this with MvL, I'll check this patch into trunk and
2.5. Alec, if you find other issues, please create a new patch.

Committed revision 61458.
Committed revision 61485. (2.5)