Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I propose to add a MemoryError read-only singleton to fix this corner case. Attributes cannot be modified, new attributes cannot be added. MemoryError attributes values:

• __cause__ = None
• __context__ = None
• __suppress_context__ = False
• __traceback__ = None
• args = ()

PyException_SetTraceback(), PyException_SetCause() and PyException_SetContext() do nothing when called on the MemoryError singleton.

A MemoryError can be raised on an integer overflow when computing the size of a memory block. In this case, you may still have available free memory and so you may be able to build a traceback object, chain exceptions, etc.

If you consider that attributes must be modifiable in this case, we can keep the MemoryError unchanged and keep the free list, but add a new special "ReadOnlyMemoryError" type which has exactly one instance, preallocated at startup (the singleton).

Read-only attributes are required to not keep references to objects when the MemoryError (singleton) is ne more used ("destroyed"): see for example test_memory_error_cleanup() of test_exceptions.

Python has already a PyExc_RecursionErrorInst singleton, which is used on recursion error. It is just a preallocated RuntimeError error, attributes are modifiable. Does this exception keep a traceback when it is no more used ("destroyed")?

Maybe using read-only attributes is overkill? Replacing the free list with a singleton is enough?

--

See also issue bpo-5437: before Python had a MemoryError singleton, but it was replaced by a free list to not keep references.

changeset: 65690:c6d86439aa91
branch: 3.1
parent: 65672:e4425d68dadf
user: Antoine Pitrou <solipsis@pitrou.net>
date: Thu Oct 28 23:06:57 2010 +0000
files: Include/pyerrors.h Lib/test/test_exceptions.py Misc/NEWS Objects/exceptions.c Python/errors.c
description:
Merged revisions 85896 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
r85896 | antoine.pitrou | 2010-10-29 00:56:58 +0200 (ven., 29 oct. 2010) | 4 lines

Issue bpo-5437: A preallocated MemoryError instance should not hold traceback
data (including local variables caught in the stack trace) alive infinitely.
........

--

Another option is maybe to clear frames of the traceback. Issue bpo-17934 added a clear() method to frame objects for that.

Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I got this bug when I worked on the issue bpo-19817 which adds an arbitary limit to memory allocations using tracemalloc. I used this limit on the Python test suite to check how Python behaves under very low memory condition.

Under very low memory condition, PyErr_NoMemory() or
PyErr_NormalizeException() enters an unlimited loop when the free list
of MemoryError becomes empty.

The real question is why the free list becomes empty.

Either way, I don't think a read-only singleton is a good idea. It may be simpler to call Py_FatalError in such cases.

Victor, do you still want to champion this, or shall we close it?

Serhiy recently worked on MemoryError, maybe he wants to work on this issue?

I'm no more interested to work on this issue.