classification
Title: sub-process would be terminated when registered finalizers are still running
Type: behavior Stage:
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: mrqianjinsi
Priority: normal Keywords:

Created on 2019-06-12 08:35 by mrqianjinsi, last changed 2019-06-12 10:21 by mrqianjinsi.

Messages (1)
msg345310 - (view) Author: (mrqianjinsi) Date: 2019-06-12 08:35
Hi guys.
I'm using multiprocessing module to accelerate my program. and I want to do some cleanup work when sub-process exit. 
but I found that sub-process would be terminated when registered finalizer are working.

Is this behavior designed intentionally?

code example:

import multiprocessing as mp
from multiprocessing.util import Finalize
import os
import time

def finalizer():
    time.sleep(0.2) # some time consuming work
    print('do cleanup work: {}'.format(os.getpid()))

def worker(_):
    print('do some work: {}'.format(os.getpid()))

def initializer():
    # ref: https://github.com/python/cpython/blob/master/Lib/multiprocessing/util.py#L147
    Finalize(None, finalizer, exitpriority=1)

    # atexit module don't work along with multiprocessing module
    # because sub-process exit via os._exit()
    # ref: https://docs.python.org/3/library/atexit.html
    # atexit.register(finalizer)  # don't work

print('main process ID: {}'.format(os.getpid()))
with mp.Pool(4, initializer=initializer) as executor:
    executor.map(worker, range(20))


gist link: https://gist.github.com/MrQianJinSi/2daf5b6a9ef08b00facdfbea5200dd28
History
Date User Action Args
2019-06-12 10:21:36mrqianjinsisettitle: sub-process would be terminated when registered finalizers are working -> sub-process would be terminated when registered finalizers are still running
2019-06-12 08:37:15mrqianjinsisettitle: sub-process would be terminated when registered finalizer are working -> sub-process would be terminated when registered finalizers are working
2019-06-12 08:35:31mrqianjinsicreate