Author abn
Recipients abn
Date 2013-02-22.06:20:52
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1361514053.16.0.91883779216.issue17273@psf.upfronthosting.co.za>
In-reply-to
Content
The task/worker handler threads in the multiprocessing.pool.Pool class are (in accordance to posix standards) not copied over when the process containing the pool is forked.

This leads to a situation where the Pool keeps receiving tasks but the tasks never get handled. This could potentially lead to deadlocks if AsyncResult.wait() is called.

Not sure if this should be considered as a bug, or an invalid use case. However, this becomes a problem when importing modules that use pools and the main code uses multiprocessing too.

[BAD] Workaround:
Reassigning Pool._task_handler to a new instance of threading.Thread after the fork seems to work in the case highlighted in the example.

Environment:
Fedora 18 
Linux 3.7.8-202.fc18.x86_64 #1 SMP Fri Feb 15 17:33:07 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
python3-3.3.0-1.fc18.x86_64

An example of this issue is shown below:

from multiprocessing import Pool, Process

def t2():
	# We expect the pool to handle this
	print('t2: Hello!')

pool = Pool()
def t1():
	# We assign a task to the pool
	pool.apply_async(t2)
	print('t1: Hello!')

if __name__ == '__main__':
	# Process() forks the main process containing the pool
	Process(target=t1).start()
History
Date User Action Args
2013-02-22 06:20:53abnsetrecipients: + abn
2013-02-22 06:20:53abnsetmessageid: <1361514053.16.0.91883779216.issue17273@psf.upfronthosting.co.za>
2013-02-22 06:20:53abnlinkissue17273 messages
2013-02-22 06:20:52abncreate