Hello,

I came across with what seems like a bug (or at least a disagreement with the current documentation).

Discussion:

I expected that after creating a numpy-array (on terminal 1), which is backed by shared memory, I would be able to use it in other terminals until I would call `shm.unlink()` (on terminal 1), at which point, the memory block would be released and no longer accessible.

What happened is that after accessing the numpy-array from terminal 2, I called 'close()' on the local 'existing_shm' instance and exited the interpreter, which displayed the `warning` seen below. 

After, I tried to access the same shared memory block from terminal 3, and a FileNotFoundError was raised. (The same error was also raised when I tried to call 'shm.unlink()' on terminal 1, after calling 'close()' on terminal 2.)

It seems that calling `close()` on an instance, destroys further access to the shared memory block from any point, while what I expected was to be able to access the array (i.e. on terminal 2), modify it, "close" my access to it, and after be able to access the modified array on i.e. terminal 3.

If the error is on my side I apologize for raising this issue and I would appreciate for clarification on what I am doing wrong.

Thank you.
Diogo


Please check below for the commands issued:

## Terminal 1

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> a = np.array([x for x in range(10)])
>>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes)
>>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
>>> b[:] = a[:]
>>> 
>>> shm.name
'psm_592ec635'


## Terminal 2

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
>>> c = np.ndarray((10,), dtype=np.int64, buffer=existing_shm.buf)
>>> 
>>> c
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> 
>>> del c
>>> existing_shm.close()
>>> 
>>> exit()
~: /usr/lib/python3.8/multiprocessing/resource_tracker.py:203: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '


## Finally, on terminal 3

>>> from multiprocessing import shared_memory
>>> import numpy as np
>>> 
>>> existing_shm = shared_memory.SharedMemory('psm_592ec635')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.8/multiprocessing/shared_memory.py", line 100, in __init__
    self._fd = _posixshmem.shm_open(
FileNotFoundError: [Errno 2] No such file or directory: '/psm_592ec635'

I've run into the same problem. It appears that the SharedMemory class is assuming that all clients of a segment are child processes from a single parent, and that they inherit the same resource_tracker. If you run separate, unrelated processes, you get a separate resource_tracker for each process. Then, when a process does a close() followed by a sys.exit(), the resource tracker detects a leak and unlinks the segment.

In my application, segment names are stored on the local filesystem and a specific process is responsible for unlinking the segment when it is shut down. I was able to get this model to work with the current SharedMemory implementation by having processes which are just doing a close() also call resource_tracker.unregister() directly to prevent their local resource trackers from destroying the segment.

I imagine the documentation needs some discussion of the assumed process model and either: 1) a statement that you need to inherit the resource tracker from a parent process, 2) a blessed way to call the resource tracker to manually unregister, or 3) a way to disable the resource tracker when creating the SharedMemory object.

Follow up - tested on Linux (The first solution).

The solution presented below will fix the problem with the caveat that the base process (the one that creates the shared-memory obj) must outlive any process that use the shared-memory.
The rationale is that unless *this* process is creating a new shared-memory object (as opposed to attaching itself to an already existing one), then there is no point to register itself to be tracked. By making this small change, the problem I mentioned when I opened this issue disappears.

#----------------------------------------------------
# https://github.com/python/cpython/blob/master/Lib/multiprocessing/shared_memory.py#L116

by changing:

from .resource_tracker import register
register(self._name, "shared_memory")

# To:

if create:
    from .resource_tracker import register
    register(self._name, "shared_memory")

#----------------------------------------------------

To retain the ability for the base process to be able to exit before those that use the shared-memory obj that the base process itself created (the current/problematic implementation), as well as fix the issue, I suggest the following approach:

When (and only when) a new shared-memory obj is created, such is registered on a new class variable of the resource-tracker, hence it can always be accessed and closed/unlinked by any process later on - this differs from the current approach, where each process that wants to access the shared-memory obj is being registered on the resource-tracker.

I look forward for any discussion on the subject.

Thank you,
Diogo

Any update on this issue?

I confirm the same issue as Diogo. The provided workaround of unregistering the sharedmemory segment in the 'consuming' process, as suggested by Jeff, solves the issue where exiting the consuming process causes the tracker to incorrectly free the shared memory. Diogo's fix to shared_memory.py#L116 does just that (actually it avoids registering it in the first place) and therefor seems ok to me.

Not sure about "it can always be accessed and closed/unlinked by any process later on", as each process will be spawning its own resource_tracker, using a separate pipe. Thus, unregister calls from other processes will not have any effect. 

The documentation is indeed unclear that processes must share the resource_tracker. Can we introduce a new flag - "persist", that would indicate no resource tracking is needed? Registering will only happen if create=True and persist=False, meaning the user accepts the creating process must outlive all other processes that could connect to the shared memory. If persist=False, the user accepts the responsibility for manual cleaning up of the allocated memory. This will allow catering to a wider range of use cases, where readers/writer processes can exit and re-connect to shared_memory as they see fit.

Having a flag seems like a good solution to me. I've also encountered this issue and posted on stack overflow about it here:

https://stackoverflow.com/questions/62748654/python-3-8-shared-memory-resource-tracker-producing-unexpected-warnings-at-appli

Hi, shared_memory has lot of issues which are mainly being caused due to resource tracking. Initially resource tracking was implemented to keep track of semaphores only, but for some reason resource tracker also started to keep track of shared_memory.
This causes shared memory to be practically useless when used by unrelated processes, because it will be unlinked as soon as a process dies, by processes which are yet to be spawned.

There is already a PR open to fix this https://github.com/python/cpython/pull/15989/files , by applio(a core developer), but for some reason it hasn't been merged yet. I will try to fix the conflicts and request it to be merged.

Now, this will fix most of the issues in shared memory, but still the current implementation of shared memory for linux won't be consistent with windows (which isn't at the moment also). You can read more about the same here: https://bugs.python.org/issue38119#msg352050

@rauanargyn , persist flag won't be good idea because it cannot be supported for windows easily, since windows uses a reference counting mechanism to keep track of shared memory and frees it as soon as all the processes using it are done.

I have tried a different approach using https://gitlab.com/tenzing/shared-array and I got it to perform well on Linux. 
Basically, the code above places all numpy arrays in /dev/shm which allows you to access and modify them from any number of processes without creating any copies; for deleting is equally simple - The code provides a SharedArray.list() to list all objects that itself placed in /dev/shm and so one can just iterate over the list and delete each element. (An easier approach is to use PathLib and just unlik all shared memory objects in /dev/shm)

I guess a solution based on Mat's code could be adapted to try and solve  the shared-memory problems.

I look forward for further discussion on the subject.

Diogo

I declare this a duplicate of issue 38119.