classification
Title: Singleton MemoryError can hold traceback data and locals indefinitely
Type: resource usage Stage: resolved
Components: Versions: Python 3.1, Python 3.2
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: BreamoreBoy, alexandre.vassalotti, amaury.forgeotdarc, pitrou
Priority: normal Keywords: patch

Created on 2009-03-08 01:19 by pitrou, last changed 2010-10-28 23:07 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
memerr.py pitrou, 2009-03-08 01:19
memerror.patch pitrou, 2009-10-29 21:22
memerror.patch pitrou, 2010-10-28 17:41
Messages (10)
msg83292 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-08 01:19
The PyExc_MemoryErrorInst object is persistent and its members never get
cleared. This means any local variable which gets caught in the
traceback isn't deallocated until the next MemoryError (!). Sample
script demonstrates this.

(this doesn't seem to affect 2.x because the traceback isn't attached to
the exception instance)
msg83293 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-08 01:21
The following patch fixes the case when the exception is discarded in
Python, but not when e.g. PyErr_Clear() is used from C code.
msg83296 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-08 01:34
A proper fix would probably be to maintain a bunch of preallocated
instances in a freelist rather than relying on an explicit singleton. It
would enforce proper dereferencing and garbage collection semantics.
msg89181 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-10 02:40
Honestly, I don't think it is a big issue. MemoryErrors are rare and
typically cause the interpreter to shutdown.

By the way, do you think the static PyExc_RecursionErrorInst object is
affected by this bug?
msg94684 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-29 21:22
Here is a patch preallocating a freelist of MemoryErrors.

To Alexandre: yes, PyExc_RecursionErrorInst probably has the same problem.
msg110325 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-14 20:31
Would someone please check out thie patch, it's mostly changes to C code that I'm not qualified to comment on.
msg112001 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-29 20:41
Changing priority as nobody appears to have lost any sleep over this issue.
msg119806 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-28 17:41
Updated patch against py3k.
msg119822 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-28 22:41
I can't manage to reproduce the issue with PyExc_RecursionErrorInst. It seems this instance is only used in a very select condition, that is, when the recursion limit is hit when trying to "normalize" an exception. Therefore, I will add a test without "fixing" anything.
msg119823 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-10-28 23:07
Patch with improved tests committed in r85896 (3.2) and r85898 (3.1).
History
Date User Action Args
2010-10-28 23:07:25pitrousetstatus: open -> closed
resolution: fixed
messages: + msg119823

stage: patch review -> resolved
2010-10-28 22:41:15pitrousetmessages: + msg119822
2010-10-28 17:41:33pitrousetfiles: + memerror.patch

messages: + msg119806
2010-07-29 20:41:41BreamoreBoysetpriority: high -> normal

messages: + msg112001
2010-07-14 20:31:35BreamoreBoysetnosy: + BreamoreBoy
messages: + msg110325
2009-10-29 21:23:10pitrousetstage: needs patch -> patch review
2009-10-29 21:23:01pitrousetfiles: - issue5437.patch
2009-10-29 21:22:44pitrousetfiles: + memerror.patch

messages: + msg94684
versions: + Python 3.2, - Python 3.0
2009-06-10 02:40:46alexandre.vassalottisetnosy: + alexandre.vassalotti
messages: + msg89181
2009-03-08 01:34:57pitrousetmessages: + msg83296
2009-03-08 01:21:56pitrousetfiles: + issue5437.patch
keywords: + patch
messages: + msg83293
2009-03-08 01:19:50pitroucreate