OS: Win10 / 8.1
Python: 3.5 / 3.6

My program use mp.Array to share huge data.
But suffer from out of memory after running for a while.

But Windows task manager didn't show which process use that huge memory. And I use pympler to check my python memory usage. Still shows noting. So, I use RamMap to check, it shows a huge shared memory is used.

I can reproduce the case by the simple test code:
#-------------------------------------------------------

   import numpy as np
   import multiprocessing as mp
   import gc

   def F ():
       a = mp.Array ( 'I', 1800000000, lock = False )

F ()
gc.collect ()
#-------------------------------------------------------

No matter how hard I tried. the memory is not released.
I put what I tried in the attachment picture.

mp.heap.BufferWrapper._heap = mp.heap.Heap ()
gc.collect ()

This is working!! The memory is cleared.
Just delete the globe _heap and recreate a new one, but is the the correct way???

Ok, this is because the multiprocessing Heap object never releases any unused arena objects, so the shared memory you allocate will probably stay allocated until the process tree ends.

It is possible to change the strategy to delete unused arenas, though it's unsure whether doing so has adverse consequences (such as making later allocations costlier). It may make sense to only reclaim the larger arenas (larger than 1MB perhaps?).

It is also possible to uncommit the memory without deallocating it, making reuse potentially faster, but that requires low-level platform-specific code (e.g. madvise(MADV_DONTNEED) on Linux or DiscardVirtualMemory() on Windows).

New changeset e4679cd by Antoine Pitrou in branch 'master':
bpo-32759: Free unused arenas in multiprocessing.heap (GH-5827)
e4679cd

I pushed a fix for this in 3.8. Since the fix is a bit delicate, I'd rather not backport it. Thank you for reporting this issue!

This change looks safe to me. I was just not sure that it is enough for practical cases. What if after allocating a large buffer the rest of the new area would be used for allocating small buffers? They can keep references to the large area after freeing the large buffer. Perhaps it is worth to block marking the remainder of a large area available. This will increase the memory consumption by small percent, but will reduce the risk of prolonging the life time of large blocks.

Yes, it's a minimal effort. More sophisticated behavior would require a more sophisticated allocator.

New changeset 857d573 by Victor Stinner in branch '3.7':
bpo-39932: Fix multiprocessing test_heap() (GH-19690)
857d573