Author neologix
Recipients flox, kaifeng, neologix
Date 2011-04-17.14:39:38
SpamBayes Score 9.75364e-09
Marked as misclassified No
Message-id <1303051180.62.0.0195417236337.issue11849@psf.upfronthosting.co.za>
In-reply-to
Content
The "problem" is not with Python, but with your libc.
When a program - such as Python - returns memory, it uses the free(3) library call.
But the libc is free to either return the memory immediately to the kernel using the relevant syscall (brk, munmap), or keep it around just in case (to simplify).
It seems that RHEL5 and onwards tend to keep a lot of memory around, at least in this case (probably because of the allocation pattern).

To sum up, python is returning memory, but your libc is not.
You can force it using malloc_trim, see the attached patch (I'm not at all suggesting its inclusion, it's just an illustration).

Results with current code:

*** Python 3.3.0 alpha
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0 29823 pts/0    S+     0:00      1  1607 168176 8596  0.2 ./python /tmp/issue11849_test.py
  1 29823 pts/0    S+     0:01      1  1607 249400 87088  2.2 ./python /tmp/issue11849_test.py
  2 29823 pts/0    S+     0:03      1  1607 324080 161704  4.1 ./python /tmp/issue11849_test.py
  3 29823 pts/0    S+     0:04      1  1607 398960 235036  5.9 ./python /tmp/issue11849_test.py
  4 29823 pts/0    S+     0:06      1  1607 473356 309464  7.8 ./python /tmp/issue11849_test.py
  5 29823 pts/0    S+     0:07      1  1607 548120 384624  9.8 ./python /tmp/issue11849_test.py
  6 29823 pts/0    S+     0:09      1  1607 622884 458332 11.6 ./python /tmp/issue11849_test.py
  7 29823 pts/0    S+     0:10      1  1607 701864 535736 13.6 ./python /tmp/issue11849_test.py
  8 29823 pts/0    S+     0:12      1  1607 772440 607988 15.5 ./python /tmp/issue11849_test.py
  9 29823 pts/0    S+     0:13      1  1607 851156 685384 17.4 ./python /tmp/issue11849_test.py
END 29823 pts/0    S+     0:16      1  1607 761712 599400 15.2 ./python /tmp/issue11849_test.py
 GC 29823 pts/0    S+     0:16      1  1607 680900 519280 13.2 ./python /tmp/issue11849_test.py
*** 29823 pts/0    S+     0:16      1  1607 680900 519288 13.2 ./python /tmp/issue11849_test.py


Results with the malloc_trim:

*** Python 3.3.0 alpha
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0 30020 pts/0    S+     0:00      1  1607 168180 8596  0.2 ./python /tmp/issue11849_test.py
  1 30020 pts/0    S+     0:01      1  1607 249404 86160  2.1 ./python /tmp/issue11849_test.py
  2 30020 pts/0    S+     0:03      1  1607 324084 160596  4.0 ./python /tmp/issue11849_test.py
  3 30020 pts/0    S+     0:04      1  1607 398964 235036  5.9 ./python /tmp/issue11849_test.py
  4 30020 pts/0    S+     0:06      1  1607 473360 309808  7.9 ./python /tmp/issue11849_test.py
  5 30020 pts/0    S+     0:07      1  1607 548124 383896  9.7 ./python /tmp/issue11849_test.py
  6 30020 pts/0    S+     0:09      1  1607 622888 458716 11.7 ./python /tmp/issue11849_test.py
  7 30020 pts/0    S+     0:10      1  1607 701868 536124 13.6 ./python /tmp/issue11849_test.py
  8 30020 pts/0    S+     0:12      1  1607 772444 607212 15.4 ./python /tmp/issue11849_test.py
  9 30020 pts/0    S+     0:14      1  1607 851160 684608 17.4 ./python /tmp/issue11849_test.py
END 30020 pts/0    S+     0:16      1  1607 761716 599524 15.3 ./python /tmp/issue11849_test.py
 GC 30020 pts/0    S+     0:16      1  1607 680776 10744  0.2 ./python /tmp/issue11849_test.py
*** 30020 pts/0    S+     0:16      1  1607 680776 10752  0.2 ./python /tmp/issue11849_test.py
History
Date User Action Args
2011-04-17 14:39:40neologixsetrecipients: + neologix, flox, kaifeng
2011-04-17 14:39:40neologixsetmessageid: <1303051180.62.0.0195417236337.issue11849@psf.upfronthosting.co.za>
2011-04-17 14:39:39neologixlinkissue11849 messages
2011-04-17 14:39:38neologixcreate