Author neologix
Recipients flox, kaifeng, neologix, pitrou
Date 2011-04-24.22:58:43
SpamBayes Score 4.23647e-08
Marked as misclassified No
Message-id <1303685925.24.0.292984878452.issue11849@psf.upfronthosting.co.za>
In-reply-to
Content
This is definitely a malloc bug.
Test with default malloc on a Debian box:

cf@neobox:~/cpython$ ./python ../issue11849_test.py 
*** Python 3.3.0 alpha
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0  3778 pts/2    S+     0:00      1  1790  8245  7024  0.5 ./python ../issue11849_test.py
  1  3778 pts/2    S+     0:17      1  1790 61937 60404  4.6 ./python ../issue11849_test.py
  2  3778 pts/2    S+     0:35      1  1790 110841 108300  8.3 ./python ../issue11849_test.py
  3  3778 pts/2    S+     0:53      1  1790 159885 158540 12.2 ./python ../issue11849_test.py
  4  3778 pts/2    S+     1:10      1  1790 209369 206724 15.9 ./python ../issue11849_test.py
  5  3778 pts/2    S+     1:28      1  1790 258505 255956 19.7 ./python ../issue11849_test.py
  6  3778 pts/2    S+     1:46      1  1790 307669 304964 23.5 ./python ../issue11849_test.py
  7  3778 pts/2    S+     2:02      1  1790 360705 356952 27.5 ./python ../issue11849_test.py
  8  3778 pts/2    S+     2:21      1  1790 405529 404172 31.2 ./python ../issue11849_test.py
  9  3778 pts/2    S+     2:37      1  1790 458789 456128 35.2 ./python ../issue11849_test.py
END  3778 pts/2    S+     3:00      1  1790 504189 501624 38.7 ./python ../issue11849_test.py
 GC  3778 pts/2    S+     3:01      1  1790 454689 453476 35.0 ./python ../issue11849_test.py
***  3778 pts/2    S+     3:01      1  1790 454689 453480 35.0 ./python ../issue11849_test.py
[56426 refs]


The heap is not trimmed, even after GC collection.
Now, using a smaller mmap threshold so that malloc uses mmap instead of brk:

cf@neobox:~/cpython$ MALLOC_MMAP_THRESHOLD_=1024 ./python ../issue11849_test.py 
*** Python 3.3.0 alpha
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0  3843 pts/2    S+     0:00      1  1790  8353  7036  0.5 ./python ../issue11849_test.py
  1  3843 pts/2    S+     0:17      1  1790 62593 59240  4.5 ./python ../issue11849_test.py
  2  3843 pts/2    S+     0:35      1  1790 112321 108304  8.3 ./python ../issue11849_test.py
  3  3843 pts/2    S+     0:53      1  1790 162313 157372 12.1 ./python ../issue11849_test.py
  4  3843 pts/2    S+     1:11      1  1790 212057 206456 15.9 ./python ../issue11849_test.py
  5  3843 pts/2    S+     1:29      1  1790 261749 255484 19.7 ./python ../issue11849_test.py
  6  3843 pts/2    S+     1:47      1  1790 311669 304484 23.5 ./python ../issue11849_test.py
  7  3843 pts/2    S+     2:03      1  1790 365485 356488 27.5 ./python ../issue11849_test.py
  8  3843 pts/2    S+     2:22      1  1790 411341 402568 31.1 ./python ../issue11849_test.py
  9  3843 pts/2    S+     2:38      1  1790 465141 454552 35.1 ./python ../issue11849_test.py
END  3843 pts/2    S+     3:02      1  1790 67173 63892  4.9 ./python ../issue11849_test.py
 GC  3843 pts/2    S+     3:03      1  1790  9925  8664  0.6 ./python ../issue11849_test.py
***  3843 pts/2    S+     3:03      1  1790  9925  8668  0.6 ./python ../issue11849_test.py
[56428 refs]

Just to be sure, with ptmalloc3 malloc implementation:

cf@neobox:~/cpython$ LD_PRELOAD=../ptmalloc3/libptmalloc3.so ./python ../issue11849_test.py 
*** Python 3.3.0 alpha
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
  0  3898 pts/2    S+     0:00      1  1790  8369  7136  0.5 ./python ../issue11849_test.py
  1  3898 pts/2    S+     0:17      1  1790 62825 60264  4.6 ./python ../issue11849_test.py
  2  3898 pts/2    S+     0:34      1  1790 112641 110176  8.5 ./python ../issue11849_test.py
  3  3898 pts/2    S+     0:52      1  1790 162689 160048 12.3 ./python ../issue11849_test.py
  4  3898 pts/2    S+     1:09      1  1790 212285 209732 16.2 ./python ../issue11849_test.py
  5  3898 pts/2    S+     1:27      1  1790 261881 259460 20.0 ./python ../issue11849_test.py
  6  3898 pts/2    S+     1:45      1  1790 311929 309332 23.9 ./python ../issue11849_test.py
  7  3898 pts/2    S+     2:01      1  1790 365625 362004 27.9 ./python ../issue11849_test.py
  8  3898 pts/2    S+     2:19      1  1790 411445 408812 31.5 ./python ../issue11849_test.py
  9  3898 pts/2    S+     2:35      1  1790 465205 461536 35.6 ./python ../issue11849_test.py
END  3898 pts/2    S+     2:58      1  1790 72141 69688  5.3 ./python ../issue11849_test.py
 GC  3898 pts/2    S+     2:59      1  1790 15001 13748  1.0 ./python ../issue11849_test.py
***  3898 pts/2    S+     2:59      1  1790 15001 13752  1.0 ./python ../issue11849_test.py
[56428 refs]

So the problem is really that glibc/eglibc malloc implementations don't automatically trim memory upon free (this happens if you're only allocating/deallocating small chunks < 64B that come from fastbins, but that's not the case here).
By the way, I noticed that dictionnaries are never allocated through pymalloc, since a new dictionnary takes more than 256B...
History
Date User Action Args
2011-04-24 22:58:45neologixsetrecipients: + neologix, pitrou, flox, kaifeng
2011-04-24 22:58:45neologixsetmessageid: <1303685925.24.0.292984878452.issue11849@psf.upfronthosting.co.za>
2011-04-24 22:58:44neologixlinkissue11849 messages
2011-04-24 22:58:43neologixcreate