classification
Title: return more memory from unicode objects to system
Type: resource usage Stage:
Components: Interpreter Core Versions: Python 2.6, Python 2.5
process
Status: closed Resolution: accepted
Dependencies: Superseder:
Assigned To: nnorwitz Nosy List: MrJean1, alecthomas, aronacher, coderanger, gvanrossum, nnorwitz
Priority: normal Keywords: patch

Created on 2008-03-17 15:48 by nnorwitz, last changed 2008-03-18 04:21 by nnorwitz. This issue is now closed.

Files
File name Uploaded Description Edit
uni.diff nnorwitz, 2008-03-17 15:48 unicode object
Messages (7)
msg63654 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2008-03-17 15:48
This patch returns more memory to the system when doing:

  >>> x = [unicode(i) for i in xrange(1000000)]
  >>> del x

If the above code is done, the memory before and after is quite
different.  After this patch, the memory of the process as reported by
the system (like top/ps) should be approximately the same
msg63674 - (view) Author: Alec Thomas (alecthomas) Date: 2008-03-17 16:55
Hi Neal,

This seems to be a more general problem than just unicode.

eg. Tuples:

>>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]
>>> del x

And user-defined objects:

>>> class A(object):
...   def __init__(self):
...     self.x = random.random()
>>> x = [A() for i in xrange(800000)]
>>> del x

Both exhibit the same behaviour. Naively to me it seems like using the
custom allocator uniformly would fix this problem.
msg63693 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2008-03-17 17:48
On Mon, Mar 17, 2008 at 11:55 AM, Alec Thomas <report@bugs.python.org> wrote:
>
>  Alec Thomas <alec@swapoff.org> added the comment:
>
>  Hi Neal,
>
>  This seems to be a more general problem than just unicode.

Kinda, sorta. The general issue is the pattern of memory
allocation/deallocation.  In the case of

>>> x = [(1, 2, 3, 4, i) for i in xrange(800000)]

The memory that is not returned is in the integer free list.  If this
code is changed to:

>>> for x in ((1, 2, 3, 4, i) for i in xrange(800000)): pass

That doesn't hold on to any extra memory.  The problem is that holes
are left in memory and a lot of it can't always be returned to the
O/S.  It can still be re-used by python.

>  Both exhibit the same behaviour. Naively to me it seems like using the
>  custom allocator uniformly would fix this problem.

In general, we should find places (like unicode) that use PyMem_* (ie,
system malloc) and replace them with PyObject_* (pymalloc).  That
should improve the behaviour, but there will always be some allocation
patterns that will be suboptimal.  Note that using pymalloc will only
help for objects < 256 bytes.  Larger objects are delegated to the
system malloc and will still exhibit some of the bad problems.

Alec, can you find places that are using the PyMem_* interface and
create a patch to fix them.  That would be great!
msg63710 - (view) Author: Alec Thomas (alecthomas) Date: 2008-03-17 18:54
> Alec, can you find places that are using the PyMem_* interface and
> create a patch to fix them.  That would be great!

Sure thing! I'll see if I can finish it today, but if not I'll work on
it during the flight to Zurich tonight.
msg63758 - (view) Author: Jean Brouwers (MrJean1) Date: 2008-03-17 20:26
Just for the record, the enhanced profiler source files in 
<http://bugs.python.org/issue2281> and <http://bugs.python.org/issue2218> 
do replace calls to malloc and free with PyObject_MALLOC resp. _FREE.
msg63833 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2008-03-18 01:03
Looks good, Neal.  Are you hesitant to check this in?  I ran a little
test showing that indeed it gives much more memory back to the system.
msg63882 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2008-03-18 04:19
After discussing this with MvL, I'll check this patch into trunk and
2.5.  Alec, if you find other issues, please create a new patch.

Committed revision 61458.
Committed revision 61485. (2.5)
History
Date User Action Args
2008-03-18 04:21:44nnorwitzsetkeywords: patch, patch
2008-03-18 04:19:49nnorwitzsetstatus: open -> closed
assignee: nnorwitz
resolution: accepted
keywords: patch, patch
2008-03-18 04:19:09nnorwitzsetkeywords: patch, patch
messages: + msg63882
2008-03-18 01:03:51gvanrossumsetkeywords: patch, patch
nosy: + gvanrossum
messages: + msg63833
2008-03-17 21:50:28coderangersetnosy: + coderanger
2008-03-17 20:26:47MrJean1setnosy: + MrJean1
messages: + msg63758
2008-03-17 18:54:03alecthomassetmessages: + msg63710
2008-03-17 17:48:47nnorwitzsetmessages: + msg63693
2008-03-17 16:55:51alecthomassetnosy: + alecthomas
messages: + msg63674
2008-03-17 16:48:02aronachersetnosy: + aronacher
2008-03-17 15:48:22nnorwitzcreate