This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: fix bsddb memory leaks
Type: Stage:
Components: None Versions: Python 2.3
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: gregory.p.smith Nosy List: chomo, gregory.p.smith, nnorwitz
Priority: normal Keywords: patch

Created on 2004-06-06 20:51 by nnorwitz, last changed 2022-04-11 14:56 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
bsddb.patch nnorwitz, 2004-06-06 20:51 mem leak patch 1
sf-patch-967763-fix.patch gregory.p.smith, 2004-06-27 23:34 committed patch for related memleaks
Messages (10)
msg46122 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-06-06 20:51
only one of the changes fixed a leak reported by
valgrind, but i think all the changes are necessary.
msg46123 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-06-13 20:56
Logged In: YES 
user_id=33168

Greg, just wanted to make sure you saw this.  It seems like
you're the only one working on bsddb these days.
msg46124 - (view) Author: alan johnson (chomo) Date: 2004-06-14 21:11
Logged In: YES 
user_id=943591

looks like the ones needed
anyway
msg46125 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2004-06-15 00:02
Logged In: YES 
user_id=413

make_key_dbt() will never return with an error leaving a
FREE_DBT(key) necessary.

however looking in depth at the code i believe it would have
been possible to have a leak when using integer keys and
exiting early with an error.  a patch (untested) that should
fix that is attached.  i'll test it out later.

do you have any memleak test cases by any chance?
msg46126 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-06-15 01:36
Logged In: YES 
user_id=33168

I just run the regression tests.  Did you see my mail to
python-dev?  

http://mail.python.org/pipermail/python-dev/2004-June/045245.html

Here's the bsddb leaks reported by valgrind:

 4 bytes in 1 blocks are definitely lost in loss record 7 of 663
    at 0x3C01DA1D: malloc (vg_replace_malloc.c:109)
    by 0x3C928265: make_key_dbt (_bsddb.c:400)
    by 0x3C92DE4F: DBC_set_range (_bsddb.c:2911)

 8 bytes in 2 blocks are definitely lost in loss record 29
of 663
    at 0x3C01DA1D: malloc (vg_replace_malloc.c:109)
    by 0x3C928265: make_key_dbt (_bsddb.c:400)
    by 0x3C929088: DB_get (_bsddb.c:1349)
msg46127 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2004-06-15 01:47
Logged In: YES 
user_id=413

Thanks, I hadn't caught up on python-dev in a while.  What
happens if you apply my attached patch and rerun the test
under valgrind?

The 4 bytes leaked sounds exactly like what i was expecting
reading the code (an integer key causes make_key_dbt to
alloc a 32bit value).  in the DBC_set_range case without the
patch i believe it is likely that the integer is allocated
but the REALLOC flag is overridden in the old code with a
MALLOC causing BerkeleyDB to leak the old pointer and
allocate a new key for the return value (btree accessed
using an integer key).

8 bytes in 2 blocks leaked in DB_get is double what i
expected but without debug info showing the DB_get function
call arguments and database flags its hard to say what happened.
msg46128 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2004-06-16 01:48
Logged In: YES 
user_id=33168

should FREE_DBT() be called clearTxn() and friends fail? 
one memory leak is caused in DB_get() by test_recno, line 75
(d.get(100)).  if the FREE_DBT()s at line 1392-1393 are
moved outside the if, so they always execute the leak goes
away.  i'm not sure if this is really the correct solution
though.

the other leak is in test_compat, don't know anything more.
msg46129 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2004-06-27 22:51
Logged In: YES 
user_id=413

DB_get and DBC_set_range were the two leaks that showed up in valgrind using the existing testsuite.  however there were plenty more possible leak cases (most of them in odd situations like out of memory or passing a non DBTxn object in the txn parameter).  

The most common trigger for a leak in various places would be an exception exit from a many functions when using integer keys (queue & recno databases).  I have a patch which i believe fixes all of these and more after a lot of code inspection that i'll commit once my valgrind run finishes and reveals no bsddb related leaks or double frees. (ugh running valgrind -v --leak-check=yes ./python Lib/bsddb/test/test_all.py on a 5 year old laptop is slow...)

"manual memory management... how quaint."
msg46130 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2004-06-27 23:34
Logged In: YES 
user_id=413

fix committed.  Modules/_bsddb.c 1.32 and  Lib/bsddb/test/test_recno.py 1.8

patch attached.  marking as python 2.3 as this should be committed to the python 2.3 maintenance branch before 2.3.5 is released.
msg46131 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2004-12-20 00:00
Logged In: YES 
user_id=413

fixed, committed to the release23-maint branch.
History
Date User Action Args
2022-04-11 14:56:04adminsetgithub: 40356
2004-06-06 20:51:04nnorwitzcreate