classification
Title: hashlib leak on import
Type: resource usage Stage: resolved
Components: Extension Modules Versions: Python 3.6
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: christian.heimes, gregory.p.smith, petr.viktorin, thehesiod
Priority: normal Keywords:

Created on 2018-05-15 20:49 by thehesiod, last changed 2018-05-15 21:51 by thehesiod. This issue is now closed.

Messages (3)
msg316723 - (view) Author: Alexander Mohr (thehesiod) * Date: 2018-05-15 20:49
I'm seeing a lot of leaks via valgrind against the hashlib module.  It appears that it's calling OpenSSL_add_all_digests(); on init, and never calling the corresponding EVP_Cleanup on free: https://www.openssl.org/docs/man1.1.0/crypto/OpenSSL_add_all_digests.html.  I see a ton of leaks like the following:

==27765== 24 bytes in 1 blocks are still reachable in loss record 13 of 10,294
==27765==    at 0x4C28C7B: malloc (vg_replace_malloc.c:299)
==27765==    by 0xA92E337: CRYPTO_malloc (in /usr/lib64/libcrypto.so.1.0.2k)
==27765==    by 0xA9E325A: lh_insert (in /usr/lib64/libcrypto.so.1.0.2k)
==27765==    by 0xA93103E: OBJ_NAME_add (in /usr/lib64/libcrypto.so.1.0.2k)
==27765==    by 0xA9F3559: OpenSSL_add_all_digests (in /usr/lib64/libcrypto.so.1.0.2k)
==27765==    by 0xA44CF02: PyInit__hashlib (_hashopenssl.c:998)
==27765==    by 0x506E627: _PyImport_LoadDynamicModuleWithSpec (importdl.c:154)
==27765==    by 0x506DBA7: _imp_create_dynamic_impl (import.c:2008)
==27765==    by 0x5067A2A: _imp_create_dynamic (import.c.h:289)
==27765==    by 0x4F3061A: PyCFunction_Call (methodobject.c:114)
==27765==    by 0x503E10C: do_call_core (ceval.c:5074)
==27765==    by 0x5035F30: _PyEval_EvalFrameDefault (ceval.c:3377)
==27765==    by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765==    by 0x503A944: _PyEval_EvalCodeWithName (ceval.c:4139)
==27765==    by 0x503DA4D: fast_function (ceval.c:4950)
==27765==    by 0x503D3FC: call_function (ceval.c:4830)
==27765==    by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765==    by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765==    by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765==    by 0x503D922: fast_function (ceval.c:4926)
==27765==    by 0x503D3FC: call_function (ceval.c:4830)
==27765==    by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765==    by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765==    by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765==    by 0x503D922: fast_function (ceval.c:4926)
==27765==    by 0x503D3FC: call_function (ceval.c:4830)
==27765==    by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765==    by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765==    by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765==    by 0x503D922: fast_function (ceval.c:4926)
==27765==    by 0x503D3FC: call_function (ceval.c:4830)
==27765==    by 0x5035563: _PyEval_EvalFrameDefault (ceval.c:3295)
==27765==    by 0x502280F: PyEval_EvalFrameEx (ceval.c:718)
==27765==    by 0x503D70D: _PyFunction_FastCall (ceval.c:4891)
==27765==    by 0x503D922: fast_function (ceval.c:4926)

I'm not exactly sure how this is happening yet (I know the code I use does a __import__ and uses multiple threads).  It sounds like this call should be ref-counted or perhaps only done once for the life of the application.
msg316737 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2018-05-15 21:47
Thanks for notifying us. There isn't much we can do here.

Python doesn't support unloading of C extension modules. The state will persist until the process ends. It doesn't make much sense to call EVP_cleanup() on shutdown, too. It will just slow down shutdown. The memory is freed anyway once the process has terminated.
msg316738 - (view) Author: Alexander Mohr (thehesiod) * Date: 2018-05-15 21:51
closing as I'm not quite sure this is right
History
Date User Action Args
2018-05-15 21:51:22thehesiodsetstatus: open -> closed

messages: + msg316738
stage: resolved
2018-05-15 21:47:29christian.heimessetmessages: + msg316737
2018-05-15 21:41:48christian.heimessetnosy: + petr.viktorin
2018-05-15 21:38:11pitrousetnosy: + gregory.p.smith, christian.heimes
2018-05-15 20:49:31thehesiodcreate