This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: hashlib memory leak
Type: resource usage Stage:
Components: Library (Lib) Versions: Python 2.7
process
Status: closed Resolution: duplicate
Dependencies: Superseder:
Assigned To: Nosy List: ateng, jcea, vstinner
Priority: normal Keywords:

Created on 2014-03-18 14:03 by ateng, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
memoryleak_min.py ateng, 2014-03-18 14:03 demo for memory leak in linux
Messages (5)
msg213961 - (view) Author: Adrian Teng (ateng) Date: 2014-03-18 14:03
A particular usage pattern of hashlib will cause a memory leak.

This leaks:
import hashlib
import sys

if __name__ == '__main__':
    data_sha1 = "hello world"
    data_md5 = "hello world"
    for i in xrange(int(1e6)):
        hashlib.sha1(data_sha1)
        hashlib.md5(data_md5)

        if i % 1000 == 0:
            print sys.getrefcount(data_sha1), ",", sys.getrefcount(data_md5)

-------
this doesn't leak:

import hashlib
import sys


if __name__ == '__main__':
    data_sha1 = "hello world"
    data_md5 = "hello world"
    for i in xrange(int(1e6)):
        sha1 = hashlib.sha1()
        sha1.update(data_sha1)
		md5 = hashlib.md5()
		md5.update(data_md5)
        
        if i % 1000 == 0:
            print sys.getrefcount(data_sha1), ", ", sys.getrefcount(data_md5)


See attached for leak memory profiling in linux
msg213977 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-18 15:05
I'm unable to reproduce your issue with Python 2.7.5:

$ python memoryleak_min.py 
[256720896.0, 15740928.0, 139264.0]
[256724992.0, 15962112.0, 139264.0]
[256724992.0, 15966208.0, 139264.0]
[256724992.0, 15966208.0, 139264.0]
(...)
[256724992.0, 15966208.0, 139264.0]

$ python2.7 <your example>
5 , 5
5 , 5
(...)
5 , 5

What is your exact Python version? What is your OS? OS version?
msg213978 - (view) Author: Adrian Teng (ateng) Date: 2014-03-18 15:14
Python 2.7.3, Red Hat Enterprise Linux Server release 5.5, with kernal 2.6.18-308.el5
msg213981 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-03-18 15:41
"Python 2.7.3, Red Hat Enterprise Linux Server release 5.5, with kernal 2.6.18-308.el5"

So your version is older than mine. Let me check Misc/NEWS... Oh...

"Issue #15219: Fix a reference leak when hashlib.new() is called with invalid parameters."

in the "What's New in Python 2.7.4 release candidate 1" section.

Can you please try a more recent Python 2.7 version to check if it's the same bug?
msg214094 - (view) Author: Adrian Teng (ateng) Date: 2014-03-19 14:17
Yup. Tested on 2.7.5 and it doesn't leak. I guess this is a duplicate of #15219.
Cheers!
History
Date User Action Args
2022-04-11 14:58:00adminsetgithub: 65166
2014-03-19 14:18:12brett.cannonsetstatus: open -> closed
2014-03-19 14:17:12atengsetresolution: duplicate
messages: + msg214094
2014-03-18 18:01:18jceasetnosy: + jcea
2014-03-18 15:41:32vstinnersetmessages: + msg213981
2014-03-18 15:14:19atengsetmessages: + msg213978
2014-03-18 15:05:45vstinnersetnosy: + vstinner
messages: + msg213977
2014-03-18 14:03:55atengcreate