Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multiprocessing pool.py doesn't close inqueue and outqueue pipes on termination #67456

Closed
shanip mannequin opened this issue Jan 18, 2015 · 4 comments
Closed

multiprocessing pool.py doesn't close inqueue and outqueue pipes on termination #67456

shanip mannequin opened this issue Jan 18, 2015 · 4 comments
Labels
performance Performance or resource usage stdlib Python modules in the Lib dir

Comments

@shanip
Copy link
Mannequin

shanip mannequin commented Jan 18, 2015

BPO 23267
Nosy @pitrou, @MojoVampire, @applio, @zhangyangyu
Superseder
  • bpo-30966: Add multiprocessing.queues.SimpleQueue.close()
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-07-19.10:21:12.970>
created_at = <Date 2015-01-18.13:33:39.156>
labels = ['library', 'performance']
title = "multiprocessing pool.py doesn't close inqueue and outqueue pipes on termination"
updated_at = <Date 2017-07-19.10:21:12.967>
user = 'https://bugs.python.org/shanip'

    bugs.python.org fields:

    activity = <Date 2017-07-19.10:21:12.967>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2017-07-19.10:21:12.970>
closer = 'pitrou'
components = ['Library (Lib)']
creation = <Date 2015-01-18.13:33:39.156>
creator = 'shanip'
dependencies = []
files = []
hgrepos = []
issue_num = 23267
keywords = []
message_count = 4.0
messages = ['234244', '234525', '289180', '298663']
nosy_count = 8.0
nosy_names = ['pitrou', 'neologix', 'sbt', 'josh.r', 'davin', 'shanip', 'xiang.zhang', 'Winterflower']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '30966'
type = 'resource usage'
url = 'https://bugs.python.org/issue23267'
versions = []
    @shanip
    Copy link
    Mannequin Author

    shanip mannequin commented Jan 18, 2015

    Multiprocessing pool.py gets SimpleQueue objects as inqueue and outqueue.
    when it terminates, it doesn't call the close() method of the queues' readers and writers.
    As a results, 4 file pipes leak in one pool termination.

    Expected: The pool closes reader and writer pipes of the inqueue and outqueue when it terminates.
    What did happen: the pool doesn't close the pipes. 4 pipes leak.

    @shanip shanip mannequin added stdlib Python modules in the Lib dir performance Performance or resource usage labels Jan 18, 2015
    @neologix
    Copy link
    Mannequin

    neologix mannequin commented Jan 22, 2015

    Interestingly, there is no close() method on SimpleQueue...

    @Winterflower
    Copy link
    Mannequin

    Winterflower mannequin commented Mar 7, 2017

    I did some investigating using a test script and Python 3.7.0a0

    from multiprocessing import Pool
import os
import time

def f(x):
	time.sleep(30)
	return x*x

if __name__=='__main__':
	print('Main pid {0}'.format(os.getpid()))
	p = Pool(5)
	p.map(f, [1,2,3])
	print('Returned')
	time.sleep(30)
and grepping for pipe and the parentpid in the output from lsof ( lsof | grep python.*<parentpid>.*pipe ). The pipes opened at the start of the script are still open even after the line print('Returned') is executed. I suppose this is expected because I did not call *p.close()*. All pipes are cleaned up after the parent process finishes.

    When I repeat the experiment calling p.close() after p.map returns, all that is left is the 9 pipes opened by the parent. All pipes are cleaned up after parent script exits.

    @shani - could you please clarify how you were able to detect the leaking pipes?

    @pitrou
    Copy link
    Member

    pitrou commented Jul 19, 2017

    I think this issue is mistaken. The reader and writer objects are closed automatically when they are destroyed (see Connection.__del__). The only thing that may lack is a way to close them more eagerly.

    In any case, I'm closing as a duplicate of bpo-30966.

    @pitrou pitrou closed this as completed Jul 19, 2017
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    performance Performance or resource usage stdlib Python modules in the Lib dir
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @pitrou