Title: resource tracker destroys shared memory segments when other processes should still have valid access
Type: behavior Stage: patch review
Components: Library (Lib) Versions: Python 3.9, Python 3.8
Status: open Resolution:
Dependencies: Superseder:
Assigned To: davin Nosy List: davin, pablogsal, pitrou, vinay0410, vstinner
Priority: normal Keywords: patch

Created on 2019-09-11 15:58 by davin, last changed 2019-09-12 05:11 by vinay0410.

Pull Requests
URL Status Linked Edit
PR 15989 open davin, 2019-09-11 16:20
Messages (2)
msg351960 - (view) Author: Davin Potts (davin) * (Python committer) Date: 2019-09-11 15:58
The resource tracker currently destroys (via _posixshmem.shm_unlink) shared memory segments on posix systems when any independently created Python process with a handle on a shared memory segment exits (gracefully or otherwise).  This breaks the expected cross-platform behavior that a shared memory segment persists at least as long as any running process has a handle on that segment.

As described with an example scenario in issue37754:
Let's say a three processes P1, P2 and P3 are trying to communicate using shared memory.
 --> P1 creates the shared memory block, and waits for P2 and P3 to access it.
 --> P2 starts and attaches this shared memory segment, writes some data to it and exits.
 --> Now in case of Unix, shm_unlink is called as soon as P2 exits. (This is by action of the resource tracker.)
 --> Now, P3 starts and tries to attach the shared memory segment.
 --> P3 will not be able to attach the shared memory segment in Unix, because shm_unlink has been called on that segment.
 --> Whereas, P3 will be able to attach to the shared memory segment in Windows.

Another key scenario we expect to work but does not currently:
1. A multiprocessing.managers.SharedMemoryManager is instantiated and started in process A.
2. A shared memory segment is created using that manager in process A.
3. A serialized representation of that shared memory segment is deserialized in process B.
4. Process B does work with the shared memory segment that is also still visible to process A.
5. Process B exits cleanly.
6. Process A reads data from the shared memory segment after process B is gone.  (This currently fails.)

The SharedMemoryManager provides a flexible means for ensuring cleanup of shared memory segments.  The current resource tracker attempts to treat shared memory segments as equivalent to semaphore references, which is too narrow of an interpretation.  As such, the current resource tracker should not be attempting to enforce cleanup of shared memory segments because it breaks expected behavior and significantly limits functionality.
msg352050 - (view) Author: Vinay Sharma (vinay0410) * Date: 2019-09-12 05:11
Hi Davin,
This PR would fix the issues mentioned by you, by not prematurely unlinking the shared memory segment. And, therefore it would make shared memory useful in a lot of use cases.

But, this would still not make Unix's implementation consistent with Windows.
Windows uses a reference counting mechanism to count the number of processes using a shared memory segment. When all of them are done using it, Windows simply unlinks and frees the memory allocated to the shared memory segment.

I know that you already know this. I am commenting to find out, that what would be the next steps to fix the above inconsistency. You could see my last comment(msg351445) in issue37754, where I have listed some ways to implement the above reference counting mechanism. 

If you could have a look and see which one would be the best way, I would be happy to make a PR for it.
Date User Action Args
2019-09-12 05:11:37vinay0410setmessages: + msg352050
2019-09-11 16:20:54davinsetkeywords: + patch
stage: patch review
pull_requests: + pull_request15618
2019-09-11 15:58:26davincreate