Title: bsddb memory leak with MacPorts bdb 4.6
Type: resource usage Stage: resolved
Components: Extension Modules, macOS Versions: Python 2.7
Status: closed Resolution: third party
Dependencies: Superseder:
Assigned To: Nosy List: haypo, jcea, ned.deily, ronaldoussoren, tjhnson
Priority: normal Keywords:

Created on 2014-10-01 21:22 by tjhnson, last changed 2014-10-07 18:28 by jcea. This issue is now closed.

Messages (12)
msg228127 - (view) Author: TJ (tjhnson) Date: 2014-10-01 21:22
The following code causes memory usage to grow excessively.

import shelve

def func():
    for i in range(1000000):
        sh ='blah')

msg228135 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-01 22:39
What happens if you use xrange instead of range?

>>> range(1000000).__sizeof__()
>>> xrange(1000000).__sizeof__()
msg228145 - (view) Author: TJ (tjhnson) Date: 2014-10-02 00:42
Put it in the while loop. Same result. Memory usage grows about 1 GiB per minute.
msg228176 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-02 07:41
Sorry, I'm not able to reproduce any major memory leak, using a couple of different dbm implementations, and there is not enough information to go on.

To pursue further, you should identify:
- what platform (operating system distribution and version)
- exactly what version of Python you are using and its origins (if not provided by the operating system distribution):
    python2.7 -c 'import sys;print(sys.version)'
- a test case that shows exactly what you run and how you determine that memory is growing
- after running the test (and possibly aborting it) and the database file "blah" has been created:
    python2.7 -c 'import whichdb;print(whichdb.whichdb("blah"))'
That should show which dbm is being used.
msg228181 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-10-02 08:00
I modified the example a little bit to display the RSS memory 10 times. The RSS increases by +176 kB at the beginning and then it is stable.

I tested on Fedora 20 (Linux):'bla', 'c') creates a 'bsddb._DBWithCursor' object. Can you please give us the DBM type returned by:

$ ./python
Python 2.7.8+ (2.7:9b4673d7b046, Oct  1 2014, 00:20:22) 
>>> import anydbm
>>>'bla', 'c')
>>> type(d)
<class 'bsddb._DBWithCursor'>
>>> d.close()

The loop of your example can maybe by simplified to:'bla', 'c')

instead of

sh ='blah')
msg228235 - (view) Author: TJ (tjhnson) Date: 2014-10-02 16:12
Thanks for the help. For my linux box, I have no issue. This is Ubuntu 13.10:

    Linux localhost 3.11.0-26-generic #45-Ubuntu SMP Tue Jul 15 04:02:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

    2.7.5+ (default, Feb 27 2014, 19:37:08) 
    [GCC 4.8.1]

However, I'm still getting issues with my other box, which is Mac OS. I think I have the relevant info here (let me know if we need additional info):

    [10:55:00] ~$ sw_vers -productVersion

    [10:55:10] ~$ python -c "import sys; print sys.version"
    2.7.8 (default, Aug 21 2014, 20:13:48) 
    [GCC 4.2.1 Compatible Apple LLVM 5.1 (clang-503.0.40)]

    [10:55:57] ~$ python -c "import anydbm;d ='bla', 'c');print(type(d))"
    <class 'bsddb._DBWithCursor'>

    [10:56:05] ~$ cat
    import anydbm
    while True:
        d ='bla', 'c')

    [10:56:10] ~$ python &
    [1] 99030

    [10:56:18] ~$ ps -o pid,rss,vsz,command -p 99030
      PID    RSS      VSZ COMMAND
    99030 481928  3010600 python

    [10:56:56] ~$ ps -o pid,rss,vsz,command -p 99030
      PID    RSS      VSZ COMMAND
    99030 3985920  6516788 python

    [11:01:10] ~$ # After 5 minutes, "top" shows MEM as 13G

    [11:01:23] ~$ ps -o pid,rss,vsz,command -p 99030
      PID    RSS      VSZ COMMAND
    99030 13891152 16429156 python

Those numbers are stable on my Linux box, and ever growing on Mac.
msg228238 - (view) Author: TJ (tjhnson) Date: 2014-10-02 16:25
No issue with 3.4 on the Mac box.
msg228241 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2014-10-02 16:59
> [10:55:57] ~$ python -c "import anydbm;d ='bla', 'c');print(type(d))"
> <class 'bsddb._DBWithCursor'>

Hum, on Mac OS X, I don't have the bsddb module installed by default. How did you install it?
msg228243 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-02 17:34
Thanks for the detailed information.  I'm not able to reproduce on OS X 10.9.5 with the Python 2.7.8 which builds and links with its own copy of Sleepycat DB 4.7.25.  However, I do see the memory leak when using a MacPorts Python 2.7.8 that is linked with a MacPorts copy of Sleepycat DB 4.6.
msg228245 - (view) Author: TJ (tjhnson) Date: 2014-10-02 17:46
I should have mentioned that this a Macports environment. So python, bsddb all come from there. Looks like that might be the culprit.
msg228248 - (view) Author: Ned Deily (ned.deily) * (Python committer) Date: 2014-10-02 17:48
Yeah, and I'm not sure why they have their build pegged to bdb 4.6; they provide newer ports as well.  I think we can close this issue then.  Perhaps you could open a MacPorts issue and ask them to update their python27 port to use at least db47.
msg228284 - (view) Author: TJ (tjhnson) Date: 2014-10-03 00:06
For future visitors, here is the MacPorts issue:
Date User Action Args
2014-10-07 18:28:32jceasetnosy: + jcea
2014-10-03 00:06:09tjhnsonsetmessages: + msg228284
2014-10-02 17:49:20ned.deilysettitle: bsddb memory leak on Mac OS X 10.9 -> bsddb memory leak with MacPorts bdb 4.6
2014-10-02 17:48:54ned.deilysetstatus: open -> closed
resolution: third party
messages: + msg228248

stage: resolved
2014-10-02 17:46:26tjhnsonsetmessages: + msg228245
2014-10-02 17:34:40ned.deilysetassignee: ronaldoussoren ->
messages: + msg228243
2014-10-02 16:59:34hayposetmessages: + msg228241
2014-10-02 16:41:21serhiy.storchakasetassignee: ronaldoussoren

type: resource usage
components: + Extension Modules, macOS, - Library (Lib)
nosy: + ronaldoussoren
2014-10-02 16:25:05tjhnsonsetmessages: + msg228238
2014-10-02 16:14:23hayposettitle: Possible Memory Leak with 'shelve' -> bsddb memory leak on Mac OS X 10.9
2014-10-02 16:12:47tjhnsonsetmessages: + msg228235
2014-10-02 08:00:36hayposetnosy: + haypo
messages: + msg228181
2014-10-02 07:41:25ned.deilysetmessages: + msg228176
2014-10-02 00:42:01tjhnsonsetmessages: + msg228145
2014-10-01 22:39:26ned.deilysetnosy: + ned.deily
messages: + msg228135
2014-10-01 21:22:50tjhnsoncreate