This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: memory leak in multiprocessing.shared_memory.SharedMemory in Windows
Type: behavior Stage: patch review
Components: Library (Lib), Windows Versions: Python 3.10, Python 3.9, Python 3.8
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: ZackerySpytz, christian.heimes, eryksun, paul.moore, steve.dower, tim.golden, zach.ware
Priority: normal Keywords: patch

Created on 2020-06-05 22:47 by eryksun, last changed 2022-04-11 14:59 by admin.

Pull Requests
URL Status Linked Edit
PR 20684 open ZackerySpytz, 2020-06-06 21:38
Messages (2)
msg370794 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-06-05 22:47
mmap.mmap in Windows doesn't support an exist_ok parameter and doesn't correctly handle the combination fileno=-1, length=0, and tagname with an existing file mapping. SharedMemory has to work around these limitations. 

Part of the workaround for the create=False case requires mapping a view via MapViewOfFile in order to get the size from VirtualQuerySize, since mmap.mmap requires it (needlessly if implemented right) when fileno=-1. This mapped view never gets unmapped, which means the shared memory will never be freed until the termination of all processes that have opened it with create=False. Also, at least in a 32-bit process, this wastes precious address space.

_winapi.UnmapViewOfFile needs to be implemented. Then the temporary view can be unmapped as follows:

    self._name = name
    h_map = _winapi.OpenFileMapping(_winapi.FILE_MAP_READ, False, name)
    try:
        p_buf = _winapi.MapViewOfFile(h_map, _winapi.FILE_MAP_READ, 0, 0, 0)
    finally:
        _winapi.CloseHandle(h_map)
    try:
        size = _winapi.VirtualQuerySize(p_buf)
    finally:
        _winapi.UnmapViewOfFile(p_buf)
    self._mmap = mmap.mmap(-1, size, tagname=name)

[1]: https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-unmapviewoffile
msg370955 - (view) Author: Eryk Sun (eryksun) * (Python triager) Date: 2020-06-08 04:20
Thanks for working on the PR, Zackery. Would you be interested in working on improvements to mmap for 3.10? With support in mmap, the Windows-specific initialization of SharedMemory could be as simple as the following:

    # Windows Named Shared Memory

    while True:
        tagname = _make_filename() if name is None else name
        try:
            self._mmap = mmap.mmap(-1, size if create else 0,
                tagname, create=create)
            break
        except FileExistsError:
            if name is not None:
                raise

    self._name = tagname
    self._size = len(self._mmap)

The new mmap `create` parameter would default to None, which uses the current behavior that allows either opening or creating a file mapping with no sanity checking (e.g. a valid fd gets passed in, but it opens an unrelated file mapping via tagname). If `create` is true, and there's an existing file mapping named `tagname`, then raise FileExistsError. If `create` is false, call OpenFileMappingW instead of CreateFileMappingW. In this case, `fileno` must be -1, `length` is allowed to be 0, and `tagname` must be a non-empty string. If `length` is 0, map the entire file mapping and get the size via VirtualQuery: RegionSize.
History
Date User Action Args
2022-04-11 14:59:32adminsetgithub: 85059
2020-06-10 00:55:43eryksunsetnosy: + christian.heimes
2020-06-08 04:20:41eryksunsetmessages: + msg370955
2020-06-06 21:38:30ZackerySpytzsetkeywords: + patch
nosy: + ZackerySpytz

pull_requests: + pull_request19897
stage: needs patch -> patch review
2020-06-05 22:47:05eryksuncreate