classification
Title: Memory leaks when embedded interpreter is reinitialized
Type: Stage:
Components: Extension Modules, Interpreter Core Versions: Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Evgeniy.Stepanov, haypo, ncoghlan, pitrou
Priority: normal Keywords:

Created on 2014-04-29 11:55 by Evgeniy.Stepanov, last changed 2015-02-15 05:51 by ncoghlan.

Messages (6)
msg217513 - (view) Author: Evgeniy Stepanov (Evgeniy.Stepanov) Date: 2014-04-29 11:55
Following https://docs.python.org/2/c-api/init.html#Py_Finalize, I'm reinitializing embedded python interpreter multiple time in one process.

#include <Python.h>

void f() {
  Py_Initialize();
  PyRun_SimpleString("from time import time,ctime\n"
      "import datetime\n"
      "print 'Today is',ctime(time())\n");
  Py_Finalize();
}

int main(void) {
  for (int i = 0; i < 30; ++i)
    f();
  return 0;
}

I see 2 sources of memory leaks:

 * _once_registry is never teared down.

==20144==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 8120 byte(s) in 29 object(s) allocated from:
    #0 0x4a3e38 in malloc /code/llvm/build0/../projects/compiler-rt/lib/asan/asan_malloc_linux.cc:75
    #1 0x4e1524 in _PyObject_GC_Malloc build/Python-2.7.6/Modules/gcmodule.c:1499
    #2 0x4e16ff in _PyObject_GC_New build/Python-2.7.6/Modules/gcmodule.c:1521
    #3 0x57e581 in PyDict_New build/Python-2.7.6/Objects/dictobject.c:277
    #4 0x62b9a7 in _PyWarnings_Init build/Python-2.7.6/Python/_warnings.c:898
    #5 0x4c2985 in Py_InitializeEx build/Python-2.7.6/Python/pythonrun.c:254
    #6 0x4c22a4 in f() build/Python-2.7.6/1.cc:4
    #7 0x4c22a4 in main build/Python-2.7.6/1.cc:14


* datetime module creates a bunch of objects in its initialization function and never destroys them

Direct leak of 8120 byte(s) in 29 object(s) allocated from:
    #0 0x4a3e38 in malloc /code/llvm/build0/../projects/compiler-rt/lib/asan/asan_malloc_linux.cc:75
    #1 0x4e1524 in _PyObject_GC_Malloc build/Python-2.7.6/Modules/gcmodule.c:1499
    #2 0x4e16ff in _PyObject_GC_New build/Python-2.7.6/Modules/gcmodule.c:1521
    #3 0x57e581 in PyDict_New build/Python-2.7.6/Objects/dictobject.c:277
    #4 0x6cb976 in PyErr_NewException build/Python-2.7.6/Python/errors.c:577
    #5 0x757227 in initzipimport build/Python-2.7.6/./Modules/zipimport.c:1235
    #6 0x6e5797 in init_builtin build/Python-2.7.6/Python/import.c:1999
    #7 0x6e158d in load_module build/Python-2.7.6/Python/import.c:1928
    #8 0x6e6919 in import_submodule build/Python-2.7.6/Python/import.c:2700
    #9 0x6e5ac7 in load_next build/Python-2.7.6/Python/import.c:2515
    #10 0x6e036c in import_module_level build/Python-2.7.6/Python/import.c:2224
    #11 0x6e036c in PyImport_ImportModuleLevel build/Python-2.7.6/Python/import.c:2288
    #12 0x671e6f in builtin___import__ build/Python-2.7.6/Python/bltinmodule.c:49
    #13 0x502b40 in PyObject_Call build/Python-2.7.6/Objects/abstract.c:2529
    #14 0x502e82 in call_function_tail build/Python-2.7.6/Objects/abstract.c:2561
    #15 0x502e82 in PyObject_CallFunction build/Python-2.7.6/Objects/abstract.c:2585
    #16 0x6df7b0 in PyImport_Import build/Python-2.7.6/Python/import.c:2886
    #17 0x6db9a7 in PyImport_ImportModule build/Python-2.7.6/Python/import.c:2129
    #18 0x6db9a7 in _PyImportHooks_Init build/Python-2.7.6/Python/import.c:239
    #19 0x4c287f in Py_InitializeEx build/Python-2.7.6/Python/pythonrun.c:248
    #20 0x4c22a4 in f() build/Python-2.7.6/1.cc:4
    #21 0x4c22a4 in main build/Python-2.7.6/1.cc:14
msg217535 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-29 18:27
Thanks. It would be nice if you could try the same with Python 3.4, or the development version.
msg217540 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-04-29 19:14
Are we fixing these on a case by case basis or is it hopeless (msg146615)?
msg217541 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-29 19:20
I think fixing on a case by case is fine.
msg218517 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2014-05-14 10:48
I've run Evgeniy's example under Valgrind and changed it to import
various C extensions. Unfortunately they all leak more or less, so
perhaps we can revisit this when (if?) the PEP 3121 and PEP 384
changes have been implemented.
msg236023 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2015-02-15 05:51
For the record, the open issues about applying 3121 and 384 to the standard libary: http://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus%2Ctype&%40sort=-activity&%40filter=status&%40action=searchid&ignore=file%3Acontent&%40search_text=pep+3121&submit=search&status=-1%2C1%2C3
History
Date User Action Args
2015-02-15 05:51:28ncoghlansetnosy: + ncoghlan
messages: + msg236023
2014-10-14 15:53:23skrahsetnosy: - skrah
2014-05-14 10:48:52skrahsetmessages: + msg218517
2014-04-30 09:53:29hayposetnosy: + haypo
2014-04-29 19:20:41pitrousetmessages: + msg217541
2014-04-29 19:14:41skrahsetmessages: + msg217540
2014-04-29 18:27:34pitrousetnosy: + skrah, pitrou
messages: + msg217535
2014-04-29 11:55:37Evgeniy.Stepanovcreate