Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently #75801

Closed
zackelan mannequin opened this issue Sep 28, 2017 · 6 comments
Closed

asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently #75801

zackelan mannequin opened this issue Sep 28, 2017 · 6 comments
Labels
3.7 (EOL) end of life performance Performance or resource usage topic-asyncio

Comments

@zackelan
Copy link
Mannequin

zackelan mannequin commented Sep 28, 2017

BPO 31620
Nosy @asvetlov, @cjrh, @1st1, @SurenNihalani
PRs
  • bpo-31620: asyncio.Queue leaks memory if the queue is empty and the consumers poll it frequently #3813
  • [3.6] bpo-31620: have asyncio/queues not leak memory when you've exceptions during waiting (GH-3813) #4326
    		•
    Files
  • test.py
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2017-12-13.19:16:34.008>
created_at = <Date 2017-09-28.17:27:49.202>
labels = ['expert-asyncio', '3.7', 'performance']
title = 'asyncio.Queue leaks memory if the queue is empty and consumers poll it frequently'
updated_at = <Date 2017-12-13.19:16:34.007>
user = 'https://bugs.python.org/zackelan'

    bugs.python.org fields:

    activity = <Date 2017-12-13.19:16:34.007>
actor = 'asvetlov'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-13.19:16:34.008>
closer = 'asvetlov'
components = ['asyncio']
creation = <Date 2017-09-28.17:27:49.202>
creator = 'zackelan'
dependencies = []
files = ['47174']
hgrepos = []
issue_num = 31620
keywords = ['patch']
message_count = 6.0
messages = ['303264', '303290', '303371', '305787', '305789', '308234']
nosy_count = 5.0
nosy_names = ['asvetlov', 'cjrh', 'yselivanov', 'snihalani', 'zackelan']
pr_nums = ['3813', '4326']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue31620'
versions = ['Python 3.6', 'Python 3.7']
    @zackelan
    Copy link
    Mannequin Author

    zackelan mannequin commented Sep 28, 2017

    Repro: Call asyncio.wait_for(some_queue.get(), some_timeout) repeatedly, with no items in the queue, so that the call times out each time.

    Expected: No increase in memory while polling an empty queue

    Actual: The queue holds on to pending "getter" futures until a item passes through the queue, which clears the pending tasks out.

    Use case:

    I have producer and consumer asyncio.Tasks, linked by an asyncio.Queue. The producer is idle most of the time and pushes messages very infrequently. The consumer task polls the queue with wait_for and a timeout so that it's able to record "yep, I'm still idle" when the wait_for times out.

    Attached script has a minimal repro. The producer emits an item every 60 seconds and the consumer polls every second. At the 59th second the _getters member of the queue is holding on to 59 pending futures. By varying the intervals this can leak an arbitrary amount of memory, for example producer emitting an item once a day vs. consumer polling once a second will leak 86,400 futures.

    From the attached script:

    2017-09-28 10:09:12,699 Queue <Queue maxsize=100 _getters[1]> is idle
    ...
    2017-09-28 10:10:09,784 Queue <Queue maxsize=100 _getters[58]> is idle
    2017-09-28 10:10:10,785 Queue <Queue maxsize=100 _getters[59]> is idle
    2017-09-28 10:10:11,699 Received 0 from <Queue maxsize=100 tasks=1>
    2017-09-28 10:10:12,700 Queue <Queue maxsize=100 _getters[1]> is idle
    2017-09-28 10:10:13,702 Queue <Queue maxsize=100 _getters[2]> is idle

    Noticed this in 3.6.1, though based on no changes to https://github.com/python/cpython/blob/master/Lib/asyncio/queues.py between the 3.6 branch and master I suspect it also affects 3.6.x and 3.7.

    @zackelan zackelan mannequin added topic-asyncio performance Performance or resource usage labels Sep 28, 2017
    @SurenNihalani
    Copy link
    Mannequin

    SurenNihalani mannequin commented Sep 29, 2017

    I am working on fixing this.

    @cjrh
    Copy link
    Mannequin

    cjrh mannequin commented Sep 29, 2017

    This looks like a dupe, or at least quite closely related to https://bugs.python.org/issue26259. If the PR resolves both issues that one should be closed too.

    @asvetlov
    Copy link
    Contributor

    asvetlov commented Nov 7, 2017

    New changeset c62f0cb by Andrew Svetlov (Suren Nihalani) in branch 'master':
    bpo-31620: have asyncio/queues not leak memory when you've exceptions during waiting (bpo-3813)
    c62f0cb

    @asvetlov
    Copy link
    Contributor

    asvetlov commented Nov 7, 2017

    New changeset ac4f6d4 by Andrew Svetlov (Miss Islington (bot)) in branch '3.6':
    bpo-31620: have asyncio/queues not leak memory when you've exceptions during waiting (GH-3813) (bpo-4326)
    ac4f6d4

    @asvetlov
    Copy link
    Contributor

    Sorry for late closing the issue, PR was merged a month ago.

    @asvetlov asvetlov added the 3.7 (EOL) end of life label Dec 13, 2017
    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    3.7 (EOL) end of life performance Performance or resource usage topic-asyncio
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @asvetlov