On Mon, Mar 17, 2008 at 11:55 AM, Alec Thomas <report@bugs.python.org> wrote:
>
>  Alec Thomas <alec@swapoff.org> added the comment:
>
>  Hi Neal,
>
>  This seems to be a more general problem than just unicode.

Kinda, sorta. The general issue is the pattern of memory
allocation/deallocation.  In the case of

>>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]

The memory that is not returned is in the integer free list.  If this
code is changed to:

>>> for x in ((1, 2, 3, 4, i) for i in xrange(800000)): pass

That doesn't hold on to any extra memory.  The problem is that holes
are left in memory and a lot of it can't always be returned to the
O/S.  It can still be re-used by python.

>  Both exhibit the same behaviour. Naively to me it seems like using the
>  custom allocator uniformly would fix this problem.

In general, we should find places (like unicode) that use PyMem_* (ie,
system malloc) and replace them with PyObject_* (pymalloc).  That
should improve the behaviour, but there will always be some allocation
patterns that will be suboptimal.  Note that using pymalloc will only
help for objects < 256 bytes.  Larger objects are delegated to the
system malloc and will still exhibit some of the bad problems.

Alec, can you find places that are using the PyMem_* interface and
create a patch to fix them.  That would be great!