Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reference cycle in email.feedparser #71709

Closed
Costas mannequin opened this issue Jul 15, 2016 · 3 comments
Closed

Reference cycle in email.feedparser #71709

Costas mannequin opened this issue Jul 15, 2016 · 3 comments
Labels
performance Performance or resource usage topic-email

Comments

@Costas
Copy link
Mannequin

Costas mannequin commented Jul 15, 2016

BPO 27522
Nosy @warsaw, @bitdancer
Files
  • emailfeedparsergc.patch
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2016-07-16.01:31:42.564>
created_at = <Date 2016-07-15.19:17:54.962>
labels = ['expert-email', 'performance']
title = 'Reference cycle in email.feedparser'
updated_at = <Date 2016-07-16.01:31:58.416>
user = 'https://bugs.python.org/Costas'

    bugs.python.org fields:

    activity = <Date 2016-07-16.01:31:58.416>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2016-07-16.01:31:42.564>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2016-07-15.19:17:54.962>
creator = 'Costas'
dependencies = []
files = ['43733']
hgrepos = []
issue_num = 27522
keywords = ['patch']
message_count = 3.0
messages = ['270503', '270527', '270528']
nosy_count = 4.0
nosy_names = ['barry', 'r.david.murray', 'python-dev', 'Costas']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue27522'
versions = ['Python 3.5', 'Python 3.6']
    @Costas
    Copy link
    Mannequin Author

    Costas mannequin commented Jul 15, 2016

    Background: We have two (clusters of) servers running a high-throughput websocket application. One runs under Python 2.7, the other has been migrated to 3.4. Very similar code runs on both servers, and both manually call the gc periodically to reduce latency.

    Problem: We noticed that under load the 3.4 server seems to slowly "leak" memory between gc collections, while the 2.7 server doesn't have this issue.

    Investigation: I tracked down the source of all the garbage to email.feedparser, which is used by http.server to parse the headers of incoming requests. The initialiser of the FeedParser class creates a reference cycle by putting a lambda function into a member variable. The lambda contains a closure with its surrounding scope that points to the object, and the object points back to the lambda.

    Proposed fix: The lambda can be harmlessly replaced by a boolean value, which doesn't cause the problem, and it's much more readable too. I'm attaching a little patch.

    @Costas Costas mannequin added topic-email performance Performance or resource usage labels Jul 15, 2016
    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jul 16, 2016

    New changeset 6c7fd035bce3 by R David Murray in branch '3.5':
    bpo-27522: break unintended cycle in feedparser.
    https://hg.python.org/cpython/rev/6c7fd035bce3

    New changeset e1278508f3cb by R David Murray in branch 'default':
    Merge: bpo-27522: break unintended cycle in feedparser.
    https://hg.python.org/cpython/rev/e1278508f3cb

    @bitdancer
    Copy link
    Member

    Thanks, Costas.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Assignees
    No one assigned
    Labels
    performance Performance or resource usage topic-email
    Projects
    None yet
    Milestone
    No milestone
    Development

    No branches or pull requests
    1 participant
    @bitdancer