Message134358
This is definitely a malloc bug.
Test with default malloc on a Debian box:
cf@neobox:~/cpython$ ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3778 pts/2 S+ 0:00 1 1790 8245 7024 0.5 ./python ../issue11849_test.py
1 3778 pts/2 S+ 0:17 1 1790 61937 60404 4.6 ./python ../issue11849_test.py
2 3778 pts/2 S+ 0:35 1 1790 110841 108300 8.3 ./python ../issue11849_test.py
3 3778 pts/2 S+ 0:53 1 1790 159885 158540 12.2 ./python ../issue11849_test.py
4 3778 pts/2 S+ 1:10 1 1790 209369 206724 15.9 ./python ../issue11849_test.py
5 3778 pts/2 S+ 1:28 1 1790 258505 255956 19.7 ./python ../issue11849_test.py
6 3778 pts/2 S+ 1:46 1 1790 307669 304964 23.5 ./python ../issue11849_test.py
7 3778 pts/2 S+ 2:02 1 1790 360705 356952 27.5 ./python ../issue11849_test.py
8 3778 pts/2 S+ 2:21 1 1790 405529 404172 31.2 ./python ../issue11849_test.py
9 3778 pts/2 S+ 2:37 1 1790 458789 456128 35.2 ./python ../issue11849_test.py
END 3778 pts/2 S+ 3:00 1 1790 504189 501624 38.7 ./python ../issue11849_test.py
GC 3778 pts/2 S+ 3:01 1 1790 454689 453476 35.0 ./python ../issue11849_test.py
*** 3778 pts/2 S+ 3:01 1 1790 454689 453480 35.0 ./python ../issue11849_test.py
[56426 refs]
The heap is not trimmed, even after GC collection.
Now, using a smaller mmap threshold so that malloc uses mmap instead of brk:
cf@neobox:~/cpython$ MALLOC_MMAP_THRESHOLD_=1024 ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3843 pts/2 S+ 0:00 1 1790 8353 7036 0.5 ./python ../issue11849_test.py
1 3843 pts/2 S+ 0:17 1 1790 62593 59240 4.5 ./python ../issue11849_test.py
2 3843 pts/2 S+ 0:35 1 1790 112321 108304 8.3 ./python ../issue11849_test.py
3 3843 pts/2 S+ 0:53 1 1790 162313 157372 12.1 ./python ../issue11849_test.py
4 3843 pts/2 S+ 1:11 1 1790 212057 206456 15.9 ./python ../issue11849_test.py
5 3843 pts/2 S+ 1:29 1 1790 261749 255484 19.7 ./python ../issue11849_test.py
6 3843 pts/2 S+ 1:47 1 1790 311669 304484 23.5 ./python ../issue11849_test.py
7 3843 pts/2 S+ 2:03 1 1790 365485 356488 27.5 ./python ../issue11849_test.py
8 3843 pts/2 S+ 2:22 1 1790 411341 402568 31.1 ./python ../issue11849_test.py
9 3843 pts/2 S+ 2:38 1 1790 465141 454552 35.1 ./python ../issue11849_test.py
END 3843 pts/2 S+ 3:02 1 1790 67173 63892 4.9 ./python ../issue11849_test.py
GC 3843 pts/2 S+ 3:03 1 1790 9925 8664 0.6 ./python ../issue11849_test.py
*** 3843 pts/2 S+ 3:03 1 1790 9925 8668 0.6 ./python ../issue11849_test.py
[56428 refs]
Just to be sure, with ptmalloc3 malloc implementation:
cf@neobox:~/cpython$ LD_PRELOAD=../ptmalloc3/libptmalloc3.so ./python ../issue11849_test.py
*** Python 3.3.0 alpha
--- PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
0 3898 pts/2 S+ 0:00 1 1790 8369 7136 0.5 ./python ../issue11849_test.py
1 3898 pts/2 S+ 0:17 1 1790 62825 60264 4.6 ./python ../issue11849_test.py
2 3898 pts/2 S+ 0:34 1 1790 112641 110176 8.5 ./python ../issue11849_test.py
3 3898 pts/2 S+ 0:52 1 1790 162689 160048 12.3 ./python ../issue11849_test.py
4 3898 pts/2 S+ 1:09 1 1790 212285 209732 16.2 ./python ../issue11849_test.py
5 3898 pts/2 S+ 1:27 1 1790 261881 259460 20.0 ./python ../issue11849_test.py
6 3898 pts/2 S+ 1:45 1 1790 311929 309332 23.9 ./python ../issue11849_test.py
7 3898 pts/2 S+ 2:01 1 1790 365625 362004 27.9 ./python ../issue11849_test.py
8 3898 pts/2 S+ 2:19 1 1790 411445 408812 31.5 ./python ../issue11849_test.py
9 3898 pts/2 S+ 2:35 1 1790 465205 461536 35.6 ./python ../issue11849_test.py
END 3898 pts/2 S+ 2:58 1 1790 72141 69688 5.3 ./python ../issue11849_test.py
GC 3898 pts/2 S+ 2:59 1 1790 15001 13748 1.0 ./python ../issue11849_test.py
*** 3898 pts/2 S+ 2:59 1 1790 15001 13752 1.0 ./python ../issue11849_test.py
[56428 refs]
So the problem is really that glibc/eglibc malloc implementations don't automatically trim memory upon free (this happens if you're only allocating/deallocating small chunks < 64B that come from fastbins, but that's not the case here).
By the way, I noticed that dictionnaries are never allocated through pymalloc, since a new dictionnary takes more than 256B... |
|
Date |
User |
Action |
Args |
2011-04-24 22:58:45 | neologix | set | recipients:
+ neologix, pitrou, flox, kaifeng |
2011-04-24 22:58:45 | neologix | set | messageid: <1303685925.24.0.292984878452.issue11849@psf.upfronthosting.co.za> |
2011-04-24 22:58:44 | neologix | link | issue11849 messages |
2011-04-24 22:58:43 | neologix | create | |
|