Hello,

I noticed a large memory consumption in my application.
I tracked it down to be a problem with garbage collection of generator locals.

The issue was noticed in 2.6 first. Then I reproduced it in 2.7.
The test case finds some leak in 3.3 too, it seems.

Though, I cannot reproduce on Debian Squeeze (2.6.6 deb or 2.7 from source) or Ubuntu (2.7.2+ or 3.2).

Someone on OS X might confirm the same issue.
This is python 2.7.3 installed from source (using perlbrew) and GCC 4.2.1.


The output of the script is:

$ python testiterbug.py
2.7.3 (default, May 20 2012, 19:54:58)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)]

[row for row in iterit(16777216)]
Memory usage:   9.3 MB

[row for row in iterit(8388608)]
Memory usage: 266.6 MB

I don't mean perlbrew, but homebrew (an OS X package manager to install from source).

I can reproduce this on an OSX 10.8 system, both using python 2.7 and python 3.3. The growth is significantly less using python 3.3.

What's odd is that the growth does not occur when both test_iter calls use 1<<24 as the argument (or larger values).

If I'd had to guess I'd say that the free implementation doesn't return a buffer to the system for smaller blocks and does do it for larger buffers.

BTW. I don't think this is a memory leak, the amount of memory used doesn't increase when there are more calls to test_iter(1<<23).

I've added calls to vmmap to the script (using os.system) to check my guesss.

The relevant bit after the call to test_iter(1<<24):

REGION TYPE                      VIRTUAL
===========                      =======
MALLOC                            405.9M        see MALLOC ZONE table below
MALLOC guard page                    32K
MALLOC metadata                     376K
STACK GUARD                        56.0M
Stack                              8192K
VM_ALLOCATE                           4K
__DATA                             1720K
__LINKEDIT                         53.4M
__TEXT                             12.1M
__UNICODE                           544K
shared memory                         8K
===========                      =======
TOTAL                             538.1M

                                     VIRTUAL ALLOCATION      BYTES
MALLOC ZONE                             SIZE      COUNT  ALLOCATED  % FULL
===========                          =======  =========  =========  ======
DefaultMallocZone_0x100436000         404.9M     414059     390.6M     96%
DispatchContinuations_0x1007dc000      1024K         17       1056      0%
===========                          =======  =========  =========  ======
TOTAL                                 405.9M     414076     390.6M     96%

Memory usage:  14.3 MB



And after the call to test_iter(1<<13)


REGION TYPE                      VIRTUAL
===========                      =======
MALLOC                            208.9M        see MALLOC ZONE table below
MALLOC freed, no zone             456.0M
MALLOC guard page                    32K
MALLOC metadata                     392K
STACK GUARD                        56.0M
Stack                              8192K
VM_ALLOCATE                           4K
__DATA                             1720K
__LINKEDIT                         53.4M
__TEXT                             12.1M
__UNICODE                           544K
shared memory                         8K
===========                      =======
TOTAL                             797.1M

                                     VIRTUAL ALLOCATION      BYTES
MALLOC ZONE                             SIZE      COUNT  ALLOCATED  % FULL
===========                          =======  =========  =========  ======
DefaultMallocZone_0x100436000         207.9M     209467     197.0M     94%
DispatchContinuations_0x1007dc000      1024K         17       1056      0%
===========                          =======  =========  =========  ======
TOTAL                                 208.9M     209484     197.0M     94%


Memory usage: 462.6 MB



The total amount of "memory usage" is grown by about 450 MBytes  (according to psutils). This is explained by the change in MALLOC memory, before:



MALLOC                            405.9M        see MALLOC ZONE table below
MALLOC guard page                    32K
MALLOC metadata                     376K

Total: about 406 MByte

After:


MALLOC                            208.9M        see MALLOC ZONE table below
MALLOC freed, no zone             456.0M
MALLOC guard page                    32K
MALLOC metadata                     392K

Total: about 665 MByte.


The difference is about 260 MBytes, which is significant but less than the difference according to psutil. But: the psutil difference as calculated by the script is the difference in RSS, not VSIZE (which vmmap shows).

If I change the _mem function to use the VSIZE the before and after values are:

Memory usage: 2390.4 MB
Memory usage: 2846.4 MB

Difference: about 456 MByte, which is still higher than the output of vmmap. 

Note that the same behavior can be seen by a much simpler script:

    print "Large"
    x = list(range(1<<24))
    print _mem()
    del x
    print _mem()

    print

    print "Small"
    x = list(range(1<<23))
    print _mem()
    del x
    print _mem()

(where _mem is the same function as in testiterbug.py).

The output is:

Large
Memory usage: 527.3 MB
Memory usage:  19.3 MB

Small
Memory usage: 330.6 MB
Memory usage: 147.6 MB


With this script is see the same pattern with vmmap: with the "small" list a large amount of memory is in regions of type (MALLOC freed, no zone).

All of this clearly points towards malloc not releasing allocated memory to the system. This is perfectly fine and not a bug in either Python or the system.

> The test case finds some leak in 3.3 too, it seems.

3.3 should be much better since it uses mmap() and munmap() to manage the object allocator's arenas (unless this is disabled under OS X for some reason, that is).

Here under Linux:

$ python3.3 testiterbug.py 
3.3.0b1 (default:1d811e1097ed, Jul 29 2012, 20:20:37) 
[GCC 4.5.2]

[row for row in iterit(16777216)]
Memory usage:   7.8 MB

[row for row in iterit(8388608)]
Memory usage:   7.8 MB

> All of this clearly points towards malloc not releasing allocated memory to the system.
> This is perfectly fine and not a bug in either Python or the system.

So it means there's no reliable way to measure the memory consumption of the program.
Sorry for the wrong report, if this is confirmed.

> > All of this clearly points towards malloc not releasing allocated memory to the system.
> > This is perfectly fine and not a bug in either Python or the system.
> 
> So it means there's no reliable way to measure the memory consumption of the program.

What it means is that "memory consumption" itself is a bit ill-defined.
"ps", "top" and friends will tell you the amount of memory (physical or
virtual) currently assigned by the OS to the process. It doesn't tell
you whether the memory is "actually" in use by the program.

Note that, in Python 3.3, sys._debugmallocstats() gives you
fined-grained information about the object allocator. This excludes any
blocks bigger than 512 bytes, though, since those are requested directly
using malloc() and free().

Thank you for digging into this. I close the issue.

I discover now that this kind of problem is quite common in the Mac world.

Other references:
http://news.ycombinator.com/item?id=3879194
http://www.markvanda.net/apple/mac-os-x-memory-issues/