classification
Title: Reference cycle in email.feedparser
Type: resource usage Stage: resolved
Components: email Versions: Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: Costas, barry, python-dev, r.david.murray
Priority: normal Keywords: patch

Created on 2016-07-15 19:17 by Costas, last changed 2016-07-16 01:31 by r.david.murray. This issue is now closed.

Files
File name Uploaded Description Edit
emailfeedparsergc.patch Costas, 2016-07-15 19:17 review
Messages (3)
msg270503 - (view) Author: Costas (Costas) Date: 2016-07-15 19:17
Background: We have two (clusters of) servers running a high-throughput websocket application. One runs under Python 2.7, the other has been migrated to 3.4. Very similar code runs on both servers, and both manually call the gc periodically to reduce latency. 

Problem: We noticed that under load the 3.4 server seems to slowly "leak" memory between gc collections, while the 2.7 server doesn't have this issue.

Investigation: I tracked down the source of all the garbage to email.feedparser, which is used by http.server to parse the headers of incoming requests. The initialiser of the FeedParser class creates a reference cycle by putting a lambda function into a member variable. The lambda contains a closure with its surrounding scope that points to the object, and the object points back to the lambda. 

Proposed fix: The lambda can be harmlessly replaced by a boolean value, which doesn't cause the problem, and it's much more readable too. I'm attaching a little patch.
msg270527 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-07-16 01:30
New changeset 6c7fd035bce3 by R David Murray in branch '3.5':
#27522: break unintended cycle in feedparser.
https://hg.python.org/cpython/rev/6c7fd035bce3

New changeset e1278508f3cb by R David Murray in branch 'default':
Merge: #27522: break unintended cycle in feedparser.
https://hg.python.org/cpython/rev/e1278508f3cb
msg270528 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2016-07-16 01:31
Thanks, Costas.
History
Date User Action Args
2016-07-16 01:31:58r.david.murraysetversions: + Python 3.6
2016-07-16 01:31:42r.david.murraysetstatus: open -> closed
resolution: fixed
messages: + msg270528

stage: resolved
2016-07-16 01:30:37python-devsetnosy: + python-dev
messages: + msg270527
2016-07-15 19:17:54Costascreate