This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author matthiastroffaes
Recipients christian.heimes, gregory.p.smith, gvanrossum, loewis, matthiastroffaes, pitrou, r.david.murray, rhettinger, skip.montanaro
Date 2009-08-17.11:43:09
SpamBayes Score 7.785776e-09
Marked as misclassified No
Message-id <1250509392.51.0.776676584659.issue6695@psf.upfronthosting.co.za>
In-reply-to
Content
To aid the discussion, I attach another quick patch which reports the
time spent on PyXXX_ClearFreeList calls during highest generation
garbage collection (including gc.collect()). For simplicity, the timer
uses clock() so the resolution is quite limited (appears to be 10ms on
my machine) and I don't claim that this is the best way of measuring
execution speed, but at least it gives some indication.

The patch also gives an indication at how frequently the highest
generation is collected.

Below is the result of the patch on the py3k-freelist_test.py test
script, on my machine (again, debug build). For reference, I've measured
the total time spent by the script as well, with the time command.

Summarizing, (30+70+30+30+50+70+20+110)/48420.0 = 0.0085 = 0.85% of time
is spent on freeing the freelists, in my test.

Another way to look at the data is that it roughly takes 10ms for each
100MB allocated (at least for the types of data in the script). Floats
seem to be an exception and take at least twice as long (not sure why).

Keep in mind that the test merely allocates and deallocates memory,
without doing much else, so it isn't a typical Python application.

$ time ./python py3k-freelist_test.py

Memory used (begin): 121Mb

memtest 2000000 int
====================
Memory used (peak): 297Mb
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (end): 121Mb
Unfreed memory: 0Mb

memtest 2000000 str
====================
Memory used (peak): 455Mb
cleared free lists in 30000 clock ticks (30.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 float
====================
Memory used (peak): 236Mb
cleared free lists in 70000 clock ticks (70.000000ms)
Memory used (end): 127Mb
Unfreed memory: 6Mb

memtest 2000000 int
====================
Memory used (peak): 312Mb
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 Test
====================
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (peak): 372Mb
cleared free lists in 30000 clock ticks (30.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 Test2
====================
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (peak): 361Mb
cleared free lists in 30000 clock ticks (30.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 _tuple
====================
Memory used (peak): 529Mb
cleared free lists in 50000 clock ticks (50.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 _set
====================
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (peak): 846Mb
cleared free lists in 70000 clock ticks (70.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 200000 _dict
====================
Memory used (peak): 233Mb
cleared free lists in 20000 clock ticks (20.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb

memtest 2000000 Test3
====================
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
cleared free lists in 0 clock ticks (0.000000ms)
Memory used (peak): 999Mb
cleared free lists in 110000 clock ticks (110.000000ms)
Memory used (end): 123Mb
Unfreed memory: 2Mb
cleared free lists in 0 clock ticks (0.000000ms)
[40763 refs]

real	0m50.921s
user	0m48.420s
sys	0m2.330s
History
Date User Action Args
2009-08-17 11:43:12matthiastroffaessetrecipients: + matthiastroffaes, gvanrossum, loewis, skip.montanaro, rhettinger, gregory.p.smith, pitrou, christian.heimes, r.david.murray
2009-08-17 11:43:12matthiastroffaessetmessageid: <1250509392.51.0.776676584659.issue6695@psf.upfronthosting.co.za>
2009-08-17 11:43:11matthiastroffaeslinkissue6695 messages
2009-08-17 11:43:10matthiastroffaescreate