Message342201
Shared memory segments are now tracked by the brand new resource_tracker!
Thanks Antoine for the review.
Does anyone have an opinion on introducing a public API for users to make the
resource_tracker track resources of their choice?
What We have in mind is:
- making public the existing resource_tracker.register/unregister
- adding a new function, resource_tracker.make_trackable(resource_type,
cleanup_func), where:
* resource_type is a string (an identifier that will be used each time a
resource of the type resource_type needs tracking, via the call
resource_tracker.register(resource_name, resource_type)
* cleanup_func must be a callable taking a single string as argument (the
name of the resource that needs tracking). This function will be called
after the end of a process for reach resource of type resource_type the
process did not clean properly
Under the hood, make_trackable simply populates resource_tracker._CLEANUP_FUNCS
with new items.
Here is a simple example:
import os
import resource_tracker
import shutil
from multiprocessing import util
class ClassCreatingAFolder:
"""Class where each instance creates a temporary folder.
Each temporary folder is supposed to exist for the duration of the instance
that created it.
"""
def __init__(self, folder_name):
self.folder_name = folder_name
os.mkdir(folder_name)
# any instance normally garbage-collected should remove its folder, and
# notice the resource_tracker that its folder was correctly removed.
util.Finalize(self, ClassCreatingAFolder.cleanup, args=(folder_name,))
# If this session quits abruptly, the finalizer will not be called for
# the instances of ClassCreatingAFolder that were still alive
# before the shutdown. The resource_tracker comes into play, and removes
# the folders associated to each of these resources.
resource_tracker.register(
folder_name, # argument to shutil.rmtree
"ClassCreatingAFolder")
@staticmethod
def cleanup(folder_name):
resource_tracker.unregister(folder_name, "ClassCreatingAFolder")
shutil.rmtree(folder_name)
# Tell the resource_tracker how to cleanup resources created by
# ClassCreatingAFolder instances
resource_tracker.make_trackable("ClassCreatingAFolder", shutil.rmtree)
Typical resources that can be made trackable include memmaped objects,
temporary folders. Our use-case is joblib that has its own mmap type that we
would like to track using the semaphore_tracker.
Any thoughts? |
|
Date |
User |
Action |
Args |
2019-05-11 15:58:49 | pierreglaser | set | recipients:
+ pierreglaser, pitrou, pablogsal |
2019-05-11 15:58:49 | pierreglaser | set | messageid: <1557590329.54.0.795308311699.issue36867@roundup.psfhosted.org> |
2019-05-11 15:58:49 | pierreglaser | link | issue36867 messages |
2019-05-11 15:58:49 | pierreglaser | create | |
|