Hi all,

Olivier Grisel, Thomas Moreau and myself are currently working on increasing
the range of action of the semaphore_tracker in Python.

multiprocessing.semaphore_tracker is a little known module, that launches a
server process used to track the life cycle of semaphores created in a python
session, and potentially cleanup those semaphores after all python processes of
the session terminated. Normally, python processes cleanup semaphores they
create. This is however not not guaranteed if the processes get violently
interrupted (using for example the bash command "killall python")

A note on why the semaphore_tracker was introduced: Cleaning up semaphores
after termination is important because the system only supports a limited
number of named semaphores, and they will not be automatically removed till the
next reboot.

Now, Python 3.8 introduces shared memory segments creation. Shared memory is
another sensitive global system resource. Currently, unexpected termination of
processes that created memory segments will result in leaking those memory
segments. This can be problematic for large compute clusters with many users
and that are rebooted rarely.

For this reason, we expanded the semaphore_tracker to also track shared memory
segments, and renamed it resource_tracker. Shared memory segments get
automatically tracked by the resource tracker when they are created. This is a
first, self-contained fix. (1)

Additionally, supporting shared memory tracking led to a more generic design
for the resource_tracker. The resource_tracker can be now easily extended
to track arbitrary resource types.
A public API could potentially be exposed for users willing to track other
types.  One for example may want to add tracking for temporary folders creating
during python sessions.  Another use case is the one of joblib, which
is a widely-used parallel-computing package, and also the backend of
scikit-learn. Joblib relies heavily on memmapping. A public API could extend
the resource_tracker to track memmap-ed objects with very little code.

Therefore, this issue serves two purposes:
- referencing the semaphore_tracker enhancement mentioned in (1)
- discussing a potentially public resource_tracker API.

New changeset f22cc69b012f52882d434a5c44a004bc3aa5c33c by Antoine Pitrou (Pierre Glaser) in branch 'master':
bpo-36867: Make semaphore_tracker track other system resources (GH-13222)
https://github.com/python/cpython/commit/f22cc69b012f52882d434a5c44a004bc3aa5c33c

Shared memory segments are now tracked by the brand new resource_tracker!
Thanks Antoine for the review.

Does anyone have an opinion on introducing a public API for users to make the
resource_tracker track resources of their choice?

What We have in mind is:
- making public the existing resource_tracker.register/unregister
- adding a new function, resource_tracker.make_trackable(resource_type,
  cleanup_func), where:

    * resource_type is a string (an identifier that will be used each time a
      resource of the type resource_type needs tracking, via the call
      resource_tracker.register(resource_name, resource_type)
    * cleanup_func must be a callable taking a single string as argument (the
      name of the resource that needs tracking). This function will be called
      after the end of a process for reach resource of type resource_type the
      process did not clean properly

Under the hood, make_trackable simply populates resource_tracker._CLEANUP_FUNCS
with new items.

Here is a simple example:
	import os
	import resource_tracker
	import shutil
	from multiprocessing import util

	class ClassCreatingAFolder:
	    """Class where each instance creates a temporary folder.

	    Each temporary folder is supposed to exist for the duration of the instance
	    that created it.
	    """
	    def __init__(self, folder_name):
		self.folder_name = folder_name
		os.mkdir(folder_name)

		# any instance normally garbage-collected should remove its folder, and
		# notice the resource_tracker that its folder was correctly removed.
		util.Finalize(self, ClassCreatingAFolder.cleanup, args=(folder_name,))

		# If this session quits abruptly, the finalizer will not be called for
		# the instances of ClassCreatingAFolder that were still alive
		# before the shutdown. The resource_tracker comes into play, and removes
		# the folders associated to each of these resources.
		resource_tracker.register(
		    folder_name, # argument to shutil.rmtree
		    "ClassCreatingAFolder")

	    @staticmethod
	    def cleanup(folder_name):
		resource_tracker.unregister(folder_name, "ClassCreatingAFolder")
		shutil.rmtree(folder_name)

	# Tell the resource_tracker how to cleanup resources created by
	# ClassCreatingAFolder instances
	resource_tracker.make_trackable("ClassCreatingAFolder", shutil.rmtree)


Typical resources that can be made trackable include memmaped objects,
temporary folders. Our use-case is joblib that has its own mmap type that we
would like to track using the semaphore_tracker.

Any thoughts?

Since the commit there is a warning in CI and locally while running tests.

Travis CI : 

* https://travis-ci.org/python/cpython/jobs/531304357#L2099 (test_multiprocessing_forkserver)
* https://travis-ci.org/python/cpython/jobs/531304357#L2296 (test_multiprocessing_fork)

Before commit : 

./python.exe -X tracemalloc -m test --fail-env-changed test_multiprocessing_forkserver
Run tests sequentially
0:00:00 load avg: 4.71 [1/1] test_multiprocessing_forkserver
test_multiprocessing_forkserver passed in 2 min 21 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 21 sec
Tests result: SUCCESS

After commit f22cc69b012f52882d434a5c44a004bc3aa5c33c : 

./python.exe -X tracemalloc -m test --fail-env-changed test_multiprocessing_forkserver
Run tests sequentially
0:00:00 load avg: 4.08 [1/1] test_multiprocessing_forkserver
/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/multiprocessing/resource_tracker.py:198: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/unittest/case.py:683: ResourceWarning: unclosed file <_io.BufferedReader name=7>
  testMethod()
Object allocated at (most recent call last):
  File "/Users/karthikeyansingaravelan/stuff/python/cpython/Lib/subprocess.py", lineno 819
    self.stdout = io.open(c2pread, 'rb', bufsize)
test_multiprocessing_forkserver passed in 2 min 20 sec

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 20 sec
Tests result: SUCCESS

== Tests result: SUCCESS ==

1 test OK.

Total duration: 2 min 26 sec
Tests result: SUCCESS

Yes, one test I wrote in an unrelated commit does not unlink a memory segment. Now the ResourceTracker complains. Fixing it now.

Actually, I was properly unlinking the shared_memory segments. The warning messages are due to bad interactions between the ResourceTracker and the SharedMemoryManager object. In this particular case, it's easy to change a little bit the  problematic test to avoid the warnings. I will focus on solving those bad interactions right after.

test_shared_memory_cleaned_after_process_termination() uses time as a weak synchronization primitive:


        # killing abruptly processes holding reference to a shared memory
        # segment should not leak the given memory segment.
        p.terminate()
        p.wait()
        time.sleep(1.0)  # wait for the OS to collect the segment

        with self.assertRaises(FileNotFoundError):
            smm = shared_memory.SharedMemory(name, create=False)

Would it be possible to use a more reliable synchronization? Such test usually fail randomly.
https://pythondev.readthedocs.io/unstable_tests.html

As Victor said, the `time.sleep(1.0)` might lead to Heisen failures. I am not sure how to write proper strong synchronization in this case but we could instead go for something intermediate such as the following pattern:


        ...
        p.terminate()
        p.wait()
        for i in range(60):
            try:
                shared_memory.SharedMemory(name, create=False)
            except FileNotFoundError:
                # the OS successfully collected the segment as expected
                break

            time.sleep(1.0)  # wait for the OS to collect the segment

         else:
             raise AssertionError(f"Failed to collect shared_memory segment {name}")


What do you think?

We can do that, or maybe we can try to wait on the `resource_tracker's` pid?

I like Olivier's pattern. Maybe we can slowly increase the sleep to stop shortly if the resource goes away shortly.

deadline = time.monotonic() + 60.0
sleep = 0.010
while ...:
   if ....: break
   if time.monotonic() > deadline: ... assert error ...
   sleep = min(sleep * 2, 5.0)
   time.sleep(1.0)

It's kind of a common pattern. Maybe it should be an helper in test.support module.


> We can do that, or maybe we can try to wait on the `resource_tracker's` pid?

I prefer to make sure that the resource goes away without inspecting multiprocessing internals.

New changeset 50466c66509de556a8579172f82d1abb1560d8e4 by Antoine Pitrou (Pierre Glaser) in branch 'master':
bpo-36867: DOC update multiprocessing.rst (GH-13289)
https://github.com/python/cpython/commit/50466c66509de556a8579172f82d1abb1560d8e4

New changeset b1dfcad6f0d3a52c9ac31fb9763fc7962a84b27c by Antoine Pitrou (Pierre Glaser) in branch 'master':
bpo-36867: Create the resource_tracker before launching SharedMemoryManagers (GH-13276)
https://github.com/python/cpython/commit/b1dfcad6f0d3a52c9ac31fb9763fc7962a84b27c

New changeset cbe72d842646ded2454784679231e3d1e6252e72 by Victor Stinner (Pierre Glaser) in branch 'master':
bpo-36867: _test_multiprocessing: avoid weak sync primitive (GH-13292)
https://github.com/python/cpython/commit/cbe72d842646ded2454784679231e3d1e6252e72

It seems like all known bugs are fixed, I close again the issue. Thanks!

The new test is not reliable, see: bpo-37244 "test_multiprocessing_forkserver: test_resource_tracker() failed on  x86 Gentoo Refleaks 3.8".