classification
Title: multiprocessing pool.py doesn't close inqueue and outqueue pipes on termination
Type: resource usage Stage: resolved
Components: Library (Lib) Versions:
process
Status: closed Resolution: duplicate
Dependencies: Superseder: Add multiprocessing.queues.SimpleQueue.close()
View: 30966
Assigned To: Nosy List: Winterflower, davin, josh.r, neologix, pitrou, sbt, shanip, xiang.zhang
Priority: normal Keywords:

Created on 2015-01-18 13:33 by shanip, last changed 2017-07-19 10:21 by pitrou. This issue is now closed.

Messages (4)
msg234244 - (view) Author: shani (shanip) Date: 2015-01-18 13:33
Multiprocessing pool.py gets SimpleQueue objects as inqueue and outqueue.
when it terminates, it doesn't call the close() method of the queues' readers and writers.
As a results, 4 file pipes leak in one pool termination.

Expected: The pool closes reader and writer pipes of the inqueue and outqueue when it terminates.
What did happen: the pool doesn't close the pipes. 4 pipes leak.
msg234525 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2015-01-22 23:10
Interestingly, there is no close() method on SimpleQueue...
msg289180 - (view) Author: Camilla Montonen (Winterflower) Date: 2017-03-07 18:33
I did some investigating using a test script and Python 3.7.0a0

from multiprocessing import Pool
import os
import time

def f(x):
	time.sleep(30)
	return x*x

if __name__=='__main__':
	print('Main pid {0}'.format(os.getpid()))
	p = Pool(5)
	p.map(f, [1,2,3])
	print('Returned')
	time.sleep(30)
and grepping for pipe and the parentpid in the output from lsof ( lsof | grep python.*<parentpid>.*pipe ). The pipes opened at the start of the script are still open even after the line print('Returned') is executed. I suppose this is expected because I did not call *p.close()*. All pipes are cleaned up after the parent process finishes.

When I repeat the experiment calling p.close() after p.map returns, all that is left is the 9 pipes opened by the parent. All pipes are cleaned up after parent script exits. 

@shani - could you please clarify how you were able to detect the leaking pipes?
msg298663 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2017-07-19 10:21
I think this issue is mistaken.  The reader and writer objects are closed automatically when they are destroyed (see Connection.__del__).  The only thing that may lack is a way to close them more eagerly.

In any case, I'm closing as a duplicate of issue 30966.
History
Date User Action Args
2017-07-19 10:21:12pitrousetstatus: open -> closed

dependencies: - Add multiprocessing.queues.SimpleQueue.close()
superseder: Add multiprocessing.queues.SimpleQueue.close()

nosy: + pitrou
messages: + msg298663
resolution: duplicate
stage: resolved
2017-07-19 09:47:59xiang.zhangsetdependencies: + Add multiprocessing.queues.SimpleQueue.close()
2017-05-17 08:55:37xiang.zhangsetnosy: + xiang.zhang
2017-03-08 04:44:45josh.rsetnosy: + josh.r
2017-03-07 18:33:13Winterflowersetmessages: + msg289180
2017-02-12 20:36:24Winterflowersetnosy: + Winterflower
2015-01-22 23:10:19neologixsetnosy: + neologix
messages: + msg234525
2015-01-18 14:26:41pitrousetnosy: + sbt, davin
2015-01-18 13:33:39shanipcreate