Author mrqianjinsi
Recipients mrqianjinsi
Date 2019-06-12.08:35:31
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1560328531.7.0.625749907508.issue37242@roundup.psfhosted.org>
In-reply-to
Content
Hi guys.
I'm using multiprocessing module to accelerate my program. and I want to do some cleanup work when sub-process exit. 
but I found that sub-process would be terminated when registered finalizer are working.

Is this behavior designed intentionally?

code example:

import multiprocessing as mp
from multiprocessing.util import Finalize
import os
import time

def finalizer():
    time.sleep(0.2) # some time consuming work
    print('do cleanup work: {}'.format(os.getpid()))

def worker(_):
    print('do some work: {}'.format(os.getpid()))

def initializer():
    # ref: https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L147
    Finalize(None, finalizer, exitpriority=1)

    # atexit module don't work along with multiprocessing module
    # because sub-process exit via os._exit()
    # ref: https://docs.python.org/3/library/atexit.html
    # atexit.register(finalizer)  # don't work

print('main process ID: {}'.format(os.getpid()))
with mp.Pool(4, initializer=initializer) as executor:
    executor.map(worker, range(20))


gist link: https://gist.github.com/MrQianJinSi/2daf5b6a9ef08b00facdfbea5200dd28
History
Date User Action Args
2019-06-12 08:35:31mrqianjinsisetrecipients: + mrqianjinsi
2019-06-12 08:35:31mrqianjinsisetmessageid: <1560328531.7.0.625749907508.issue37242@roundup.psfhosted.org>
2019-06-12 08:35:31mrqianjinsilinkissue37242 messages
2019-06-12 08:35:31mrqianjinsicreate