This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: memory leaks when importing posix module
Type: Stage:
Components: Interpreter Core Versions: Python 2.4
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: gvanrossum Nosy List: facundobatista, gvanrossum, nnorwitz
Priority: low Keywords:

Created on 2002-09-23 14:27 by nnorwitz, last changed 2022-04-10 16:05 by admin. This issue is now closed.

File name Uploaded Description Edit
mem-test.c nnorwitz, 2002-09-23 14:27 call Py_Initialize/Py_Finalize in a loop
ss.diff nnorwitz, 2002-09-25 00:37 structseq mem alloc failure fixes
Messages (6)
msg12459 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-09-23 14:27
The attached program which calls
Py_Initialize/Py_Finalize in a loop demonstrates a
program which grows quite quickly.  This bug effects
2.2.1+ and 2.3.  valgrind reports that the memory is
still reachable, but it seems like a memory leak and
the process definitely grows.

Compile the program with libpython, and run (./mem-test
100000).  Make sure it can import site (which imports
posix module).  While the program is running, do a ps
and watch it grow.  If import site fails, the process
will not grow. can be as simple as import posix.
I believe the problem is related to
PyStructSequence_Fields for statfs.  But haven't
completely diagnosed the problem.  As I learn more, I
will add comments.

To simply importing or not importing site, mem-test
takes an optional 2nd argument which will
enable/disable loading of  ./mem-test 100000 1
will prevent import site.

I hope this is understandable, even though the
description wasn't clear.
msg12460 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-09-23 16:32
Logged In: YES 

This is a good use of your time.  Thanks for looking into
this! Assign back to me when you have a fix for review or a
stumbling block.
msg12461 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-09-25 00:37
Logged In: YES 

I'm mostly stuck.  I haven't solved the problem, but I do
have some more information.  Attached is a patch to
Object/structseq.c which fixes some problems if memory
allocation fails.  Also, I changed a PyInt_FromLong(1) to
Py_True for __safe_for_unpickling__.  Py_True could also be
used in compile.c::
symtable_load_symbols replacing the variable: implicit.

Some fields which I believe are leaking from the
PyTypeObject are:  tp_members, tp_dict, and tp_bases. 
However, there are more leaks.  I think all the remaining
leaks come from PyType_Ready().  For example, from
add_operators(), PyDescr_NewWrapper().  I thought DECREFing
tp_dict would free these, but it didn't seem to have any effect.

Part of the problem is how should these values be cleaned
up.  Putting cleanup in PyStructSequence_InitType would
guarantee the problem is fixed for all structseqs, but that
doesn't seem like a good idea.  This would assume the
PyTypeObject passed in is initialized.  This is true for
static variables, but not for any other variables.  If
there's a new API for cleanup, all the users of structseq
will need to use it.  Perhaps, this is only an issue in the
core.  I don't know about extension modules.

Finally, I suspect there may be more leaks outside of posix
too.  These seem to mostly come from _Py_ReadyTypes() called
in pythonrun.c::Py_Initialize().
msg12462 - (view) Author: Guido van Rossum (gvanrossum) * (Python committer) Date: 2002-12-26 15:16
Logged In: YES 

I expect I will have no time to look into this further any
time soon. But if you have fixes that clearly fix leaks,
please check them in!

(Shouldn't this be in the 2.3 group?)
msg12463 - (view) Author: Facundo Batista (facundobatista) * (Python committer) Date: 2005-03-21 18:04
Logged In: YES 

Changed to 2.3 group.
msg12464 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2006-11-21 08:21
The important parts of the patch have been applied.  Py_Initialize/Py_Finalize in a loop is known to be an issue and is documented AFAIK.  Closing.
Date User Action Args
2022-04-10 16:05:41adminsetgithub: 37206
2002-09-23 14:27:00nnorwitzcreate