classification
Title: memory not freed, aka memory leak continues...
Type: behavior Stage:
Components: Tests Versions: Python 3.7, Python 2.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Michael.Felt, serhiy.storchaka, vstinner
Priority: normal Keywords:

Created on 2018-01-12 18:56 by Michael.Felt, last changed 2018-01-15 09:06 by vstinner.

Messages (5)
msg309869 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-01-12 18:56
in issue25582 - the issue is not (yet) resolved. Perhaps this one can be closed and issue25582 reopened.

Both from python2-2.7.14 and "git master" I am getting:

michael@x071:[/data/prj/python/git/gcc-python3-3.7]./python -m unittest -v ctypes.test.test_pointers          <
test_abstract (ctypes.test.test_pointers.PointersTestCase) ... ok
test_basic (ctypes.test.test_pointers.PointersTestCase) ... ok
test_basics (ctypes.test.test_pointers.PointersTestCase) ... ok
test_bug_1467852 (ctypes.test.test_pointers.PointersTestCase) ... ok
test_c_void_p (ctypes.test.test_pointers.PointersTestCase) ... ok
test_callbacks_with_pointers (ctypes.test.test_pointers.PointersTestCase) ... ok
test_change_pointers (ctypes.test.test_pointers.PointersTestCase) ... ok
test_charpp (ctypes.test.test_pointers.PointersTestCase)
Test that a character pointer-to-pointer is correctly passed ... ok
test_from_address (ctypes.test.test_pointers.PointersTestCase) ... ok
test_other (ctypes.test.test_pointers.PointersTestCase) ... ok
test_pass_pointers (ctypes.test.test_pointers.PointersTestCase) ... ok
test_pointer_crash (ctypes.test.test_pointers.PointersTestCase) ... ok
test_pointer_type_name (ctypes.test.test_pointers.PointersTestCase) ... ok
test_pointer_type_str_name (ctypes.test.test_pointers.PointersTestCase) ... ERROR
test_pointers_bool (ctypes.test.test_pointers.PointersTestCase) ... ok

======================================================================
ERROR: test_pointer_type_str_name (ctypes.test.test_pointers.PointersTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/prj/python/git/gcc-python3-3.7/Lib/ctypes/test/test_pointers.py", line 208, in test_pointer_type_str_name
    large_string = 'T' * 2 ** 25
MemoryError

----------------------------------------------------------------------
Ran 15 tests in 0.319s

FAILED (errors=1)

+++++
Looking at the test source:

  +196      def test_pointer_type_name(self):
  +197          LargeNamedType = type('T' * 2 ** 25, (Structure,), {})
  +198          self.assertTrue(POINTER(LargeNamedType))
  +199
  +200          # to not leak references, we must clean _pointer_type_cache
  +201          from ctypes import _pointer_type_cache
  +202          del _pointer_type_cache[LargeNamedType]
  +203
  +204      def test_pointer_type_str_name(self):
  +205          large_string = 'T' * 2 ** 25
  +206          P = POINTER(large_string)
  +207          self.assertTrue(P)
  +208
  +209          # to not leak references, we must clean _pointer_type_cache
  +210          from ctypes import _pointer_type_cache
  +211          del _pointer_type_cache[id(P)]

****
After changing the exponent (** 25) to "** 23" on either line 197 OR 205 
- ALL test succeed -
After changing the exponent to " ** 24" on BOTH lines 197 and 205, all tests pass.

****
My concern is that the "patch" from issue 25582 the 
"del _pointer_type_cache[]" statement is not freeing memory.

+++++
What can I add to the test to debug!
+++++
p.s. - results are the same on AIX (5.3 and 6.1) Python2-2.7.14 and Pyhton3-3.7.X (git master). Compiler does not seem to matter (both xlc and gcc).
32-bit, default memory model (256 MByte max - aka (2 ** 28) for data and malloc. I could also try changing the memory model - but will do that only if the test, by definition, is not "free()ing" the memory used, when done.

HTH
Michael
msg309874 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-12 22:48
Issue25582 fixed a memory leak. If run the tests repeatedly every iteration leaked 100 MB of memory. If you get a MemoryError with a single iteration, this means that your machine just doesn't have enough memory for tests. Tests require around 500-600MB of memory on 32-bit platform.
msg309942 - (view) Author: Michael Felt (Michael.Felt) * Date: 2018-01-15 01:56
Thanks for the clarification. 

Being curious, is there a way to see what the size of the cache is? I want to believe, but i do not have the impression memory is being reallocated to later users. My gut feeling is that the code change permits repeated calls to a function in a way that it reuses memory it received before (rather than a memory leak of repeatedly allocating a brand new buffer). What I expected, more accurately hoped for, is that cache was either freed and/or made available for reuse. 

Since I am on my phone atm I am less successful at finding documentation. I would be very grateful for a link to the relevant information and I’ll look further, time permitting. Thx again,

Michael 

Sent from my iPhone

> On 12 Jan 2018, at 23:48, Serhiy Storchaka <report@bugs.python.org> wrote:
> 
> 
> Serhiy Storchaka <storchaka+cpython@gmail.com> added the comment:
> 
> Issue25582 fixed a memory leak. If run the tests repeatedly every iteration leaked 100 MB of memory. If you get a MemoryError with a single iteration, this means that your machine just doesn't have enough memory for tests. Tests require around 500-600MB of memory on 32-bit platform.
> 
> ----------
> nosy: +serhiy.storchaka
> 
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue32542>
> _______________________________________
>
msg309954 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-01-15 08:46
> Being curious, is there a way to see what the size of the cache is?

len(_pointer_type_cache)?
msg309955 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2018-01-15 09:06
ctypes test cases tearDown() may always clear _pointer_type_cache. Would it help?
History
Date User Action Args
2018-01-15 09:06:21vstinnersetnosy: + vstinner
messages: + msg309955
2018-01-15 08:46:04serhiy.storchakasetmessages: + msg309954
2018-01-15 01:56:41Michael.Feltsetmessages: + msg309942
2018-01-12 22:48:32serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg309874
2018-01-12 18:56:34Michael.Feltcreate