Title: multiprocessing.Array do not release shared memory
Type: resource usage Stage: resolved
Components: Windows Versions: Python 3.8
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: OO O, davin, paul.moore, pitrou, serhiy.storchaka, steve.dower, tim.golden, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2018-02-03 21:14 by OO O, last changed 2020-04-23 22:20 by vstinner. This issue is now closed.

File name Uploaded Description Edit
result.jpg OO O, 2018-02-03 21:14
Pull Requests
URL Status Linked Edit
PR 5827 merged pitrou, 2018-02-23 11:22
Messages (9)
msg311571 - (view) Author: OO O (OO O) Date: 2018-02-03 21:14
OS: Win10 / 8.1
Python: 3.5 / 3.6

My program use mp.Array to share huge data.
But suffer from out of memory after running for a while. 

But Windows task manager didn't show which process use that huge memory. And I use pympler to check my python memory usage. Still shows noting. So, I use RamMap to check, it shows a huge shared memory is used.

I can reproduce the case by the simple test code:
   import numpy as np
   import multiprocessing as mp
   import gc

   def F ():
       a = mp.Array ( 'I', 1800000000, lock = False )

   F ()
   gc.collect ()

No matter how hard I tried. the memory is not released.
I put what I tried in the attachment picture.
msg311573 - (view) Author: OO O (OO O) Date: 2018-02-03 21:54
mp.heap.BufferWrapper._heap = mp.heap.Heap ()
gc.collect ()

This is working!! The memory is cleared.
Just delete the globe _heap and recreate a new one, but is the the correct way???
msg312626 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-23 10:56
Ok, this is because the multiprocessing Heap object never releases any unused arena objects, so the shared memory you allocate will probably stay allocated until the process tree ends.

It is possible to change the strategy to delete unused arenas, though it's unsure whether doing so has adverse consequences (such as making later allocations costlier).  It may make sense to only reclaim the larger arenas (larger than 1MB perhaps?).
msg312628 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-02-23 11:01
It is also possible to uncommit the memory without deallocating it, making reuse potentially faster, but that requires low-level platform-specific code (e.g. madvise(MADV_DONTNEED) on Linux or DiscardVirtualMemory() on Windows).
msg315131 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-04-09 15:37
New changeset e4679cd644aa19f9d9df9beb1326625cf2b02c15 by Antoine Pitrou in branch 'master':
bpo-32759: Free unused arenas in multiprocessing.heap (GH-5827)
msg315132 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-04-09 15:42
I pushed a fix for this in 3.8.  Since the fix is a bit delicate, I'd rather not backport it.  Thank you for reporting this issue!
msg315133 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-04-09 16:01
This change looks safe to me. I was just not sure that it is enough for practical cases. What if after allocating a large buffer the rest of the new area would be used for allocating small buffers? They can keep references to the large area after freeing the large buffer. Perhaps it is worth to block marking the remainder of a large area available. This will increase the memory consumption by small percent, but will reduce the risk of prolonging the life time of large blocks.
msg315137 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2018-04-09 16:34
Yes, it's a minimal effort.  More sophisticated behavior would require a more sophisticated allocator.
msg367153 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-04-23 22:20
New changeset 857d573257ab150d6bea666354312a5fd284b171 by Victor Stinner in branch '3.7':
bpo-39932: Fix multiprocessing test_heap() (GH-19690)
Date User Action Args
2020-04-23 22:20:04vstinnersetnosy: + vstinner
messages: + msg367153
2018-04-09 16:34:18pitrousetmessages: + msg315137
2018-04-09 16:01:27serhiy.storchakasetmessages: + msg315133
2018-04-09 15:42:05pitrousetstatus: open -> closed
resolution: fixed
messages: + msg315132

stage: patch review -> resolved
2018-04-09 15:37:58pitrousetmessages: + msg315131
2018-02-25 16:02:13serhiy.storchakasetnosy: + serhiy.storchaka
2018-02-23 11:22:49pitrousetkeywords: + patch
stage: patch review
pull_requests: + pull_request5605
2018-02-23 11:01:59pitrousetmessages: + msg312628
2018-02-23 10:56:04pitrousetmessages: + msg312626
versions: + Python 3.8, - Python 3.5, Python 3.6
2018-02-04 06:21:47steve.dowersetnosy: + pitrou, davin
2018-02-03 21:54:08OO Osetmessages: + msg311573
2018-02-03 21:39:33OO Osetversions: + Python 3.5
2018-02-03 21:14:21OO Ocreate