classification
Title: Add a MemoryError singleton to fix an unlimited loop when the memory is exhausted
Type: Stage:
Components: Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Arfrever, neologix, pitrou, r.david.murray, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2013-11-29 17:24 by vstinner, last changed 2015-10-02 21:08 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
memerror_singleton.patch vstinner, 2013-11-29 17:24
Messages (5)
msg204742 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-29 17:24
Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I propose to add a MemoryError read-only singleton to fix this corner case. Attributes cannot be modified, new attributes cannot be added. MemoryError attributes values:

* __cause__ = None
* __context__ = None
* __suppress_context__ = False
* __traceback__ = None
* args = ()

PyException_SetTraceback(), PyException_SetCause() and PyException_SetContext() do nothing when called on the MemoryError singleton.

A MemoryError can be raised on an integer overflow when computing the size of a memory block. In this case, you may still have available free memory and so you may be able to build a traceback object, chain exceptions, etc.

If you consider that attributes must be modifiable in this case, we can keep the MemoryError unchanged and keep the free list, but add a new special "ReadOnlyMemoryError" type which has exactly one instance, preallocated at startup (the singleton).

Read-only attributes are required to not keep references to objects when the MemoryError (singleton) is ne more used ("destroyed"): see for example test_memory_error_cleanup() of test_exceptions.

Python has already a PyExc_RecursionErrorInst singleton, which is used on recursion error. It is just a preallocated RuntimeError error, attributes are modifiable. Does this exception keep a traceback when it is no more used ("destroyed")?

Maybe using read-only attributes is overkill? Replacing the free list with a singleton is enough?

--

See also issue #5437: before Python had a MemoryError singleton, but it was replaced by a free list to not keep references.

changeset:   65690:c6d86439aa91
branch:      3.1
parent:      65672:e4425d68dadf
user:        Antoine Pitrou <solipsis@pitrou.net>
date:        Thu Oct 28 23:06:57 2010 +0000
files:       Include/pyerrors.h Lib/test/test_exceptions.py Misc/NEWS Objects/exceptions.c Python/errors.c
description:
Merged revisions 85896 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r85896 | antoine.pitrou | 2010-10-29 00:56:58 +0200 (ven., 29 oct. 2010) | 4 lines

  Issue #5437: A preallocated MemoryError instance should not hold traceback
  data (including local variables caught in the stack trace) alive infinitely.
........

--

Another option is maybe to clear frames of the traceback. Issue #17934 added a clear() method to frame objects for that.
msg204744 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-29 17:27
> Under very low memory condition, PyErr_NoMemory() or PyErr_NormalizeException() enters an unlimited loop when the free list of MemoryError becomes empty.

I got this bug when I worked on the issue #19817 which adds an arbitary limit to memory allocations using tracemalloc. I used this limit on the Python test suite to check how Python behaves under very low memory condition.
msg204745 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-11-29 17:31
> Under very low memory condition, PyErr_NoMemory() or
> PyErr_NormalizeException() enters an unlimited loop when the free list 
> of MemoryError becomes empty.

The real question is why the free list becomes empty.

Either way, I don't think a read-only singleton is a good idea. It may be simpler to call Py_FatalError in such cases.
msg240340 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2015-04-09 15:36
Victor, do you still want to champion this, or shall we close it?
msg240353 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-04-09 17:07
Serhiy recently worked on MemoryError, maybe he wants to work on this issue?

I'm no more interested to work on this issue.
History
Date User Action Args
2015-10-02 21:08:07vstinnersetstatus: open -> closed
resolution: out of date
2015-04-09 17:07:35vstinnersetnosy: + serhiy.storchaka
messages: + msg240353
2015-04-09 15:36:07r.david.murraysetnosy: + r.david.murray
messages: + msg240340
2013-12-03 21:15:00vstinnerlinkissue19817 dependencies
2013-12-02 01:14:27alexandre.vassalottisetnosy: - alexandre.vassalotti
2013-11-29 21:51:34Arfreversetnosy: + Arfrever
2013-11-29 17:31:38pitrousetnosy: + neologix
messages: + msg204745
2013-11-29 17:27:30vstinnersetmessages: + msg204744
2013-11-29 17:24:51vstinnersetnosy: + pitrou, alexandre.vassalotti
2013-11-29 17:24:02vstinnercreate