Author pierreglaser
Recipients pablogsal, pierreglaser, pitrou
Date 2019-05-11.15:58:49
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1557590329.54.0.795308311699.issue36867@roundup.psfhosted.org>
In-reply-to
Content
Shared memory segments are now tracked by the brand new resource_tracker!
Thanks Antoine for the review.

Does anyone have an opinion on introducing a public API for users to make the
resource_tracker track resources of their choice?

What We have in mind is:
- making public the existing resource_tracker.register/unregister
- adding a new function, resource_tracker.make_trackable(resource_type,
  cleanup_func), where:

    * resource_type is a string (an identifier that will be used each time a
      resource of the type resource_type needs tracking, via the call
      resource_tracker.register(resource_name, resource_type)
    * cleanup_func must be a callable taking a single string as argument (the
      name of the resource that needs tracking). This function will be called
      after the end of a process for reach resource of type resource_type the
      process did not clean properly

Under the hood, make_trackable simply populates resource_tracker._CLEANUP_FUNCS
with new items.

Here is a simple example:
	import os
	import resource_tracker
	import shutil
	from multiprocessing import util

	class ClassCreatingAFolder:
	    """Class where each instance creates a temporary folder.

	    Each temporary folder is supposed to exist for the duration of the instance
	    that created it.
	    """
	    def __init__(self, folder_name):
		self.folder_name = folder_name
		os.mkdir(folder_name)

		# any instance normally garbage-collected should remove its folder, and
		# notice the resource_tracker that its folder was correctly removed.
		util.Finalize(self, ClassCreatingAFolder.cleanup, args=(folder_name,))

		# If this session quits abruptly, the finalizer will not be called for
		# the instances of ClassCreatingAFolder that were still alive
		# before the shutdown. The resource_tracker comes into play, and removes
		# the folders associated to each of these resources.
		resource_tracker.register(
		    folder_name, # argument to shutil.rmtree
		    "ClassCreatingAFolder")

	    @staticmethod
	    def cleanup(folder_name):
		resource_tracker.unregister(folder_name, "ClassCreatingAFolder")
		shutil.rmtree(folder_name)

	# Tell the resource_tracker how to cleanup resources created by
	# ClassCreatingAFolder instances
	resource_tracker.make_trackable("ClassCreatingAFolder", shutil.rmtree)


Typical resources that can be made trackable include memmaped objects,
temporary folders. Our use-case is joblib that has its own mmap type that we
would like to track using the semaphore_tracker.

Any thoughts?
History
Date User Action Args
2019-05-11 15:58:49pierreglasersetrecipients: + pierreglaser, pitrou, pablogsal
2019-05-11 15:58:49pierreglasersetmessageid: <1557590329.54.0.795308311699.issue36867@roundup.psfhosted.org>
2019-05-11 15:58:49pierreglaserlinkissue36867 messages
2019-05-11 15:58:49pierreglasercreate