I created a first toy based on the PEP-445 to test of Python handles MemoryError: in short, Python handles them badly...

Home page of the project:
https://pypi.python.org/pypi/pyfailmalloc

I will use this issue to track crashes found by this tool.

New changeset 111c2a070f28 by Victor Stinner in branch 'default':
Issue bpo-18408: PyObject_GC_NewVar() now raises SystemError exception if nitems
http://hg.python.org/cpython/rev/111c2a070f28

New changeset ba766323b53a by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyDict_New() to handle correctly new_keys_object() failure
http://hg.python.org/cpython/rev/ba766323b53a

New changeset 68887e177dd4 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix list.pop() to handle list_resize() failure (MemoryError).
http://hg.python.org/cpython/rev/68887e177dd4

New changeset 697d722d97f9 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix marshal reader for Unicode strings: handle
http://hg.python.org/cpython/rev/697d722d97f9

New changeset de1473f4503b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyType_Ready() and type.__bases__ setter to handle
http://hg.python.org/cpython/rev/de1473f4503b

New changeset f4311870e329 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix call_function() of ceval.c to handle PyTuple_New() failure
http://hg.python.org/cpython/rev/f4311870e329

New changeset df8b40593a08 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _PyUnicodeWriter_Finish(): clear writer->buffer,
http://hg.python.org/cpython/rev/df8b40593a08

"I created a first toy based on the PEP-445 to test of Python handles MemoryError: in short, Python handles them badly..."

Ooops: to test *how* Python handles..

New changeset ba79f6a86300 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix ConvParam() of the ctypes module to handle paramfunc failure
http://hg.python.org/cpython/rev/ba79f6a86300

New changeset 2ef2edfd1a4c by Victor Stinner in branch 'default':
Issue bpo-18408: Fix zlib.compressobj() to handle PyThread_allocate_lock() failure
http://hg.python.org/cpython/rev/2ef2edfd1a4c

New changeset a45407fa4a5b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyCode_Optimize(): raise a MemoryError on memory allocation
http://hg.python.org/cpython/rev/a45407fa4a5b

New changeset ed0c9d77e179 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix usage of _PyBytes_Resize()
http://hg.python.org/cpython/rev/ed0c9d77e179

New changeset d605c6b8095c by Victor Stinner in branch 'default':
Issue bpo-18408: _PyUnicodeWriter_Finish() now clears its buffer attribute in all
http://hg.python.org/cpython/rev/d605c6b8095c

New changeset 549d8d3297f2 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _PyMem_DebugRealloc()
http://hg.python.org/cpython/rev/549d8d3297f2

New changeset c91e7f707562 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix select.select() to handle PyList_New() failure (MemoryError)
http://hg.python.org/cpython/rev/c91e7f707562

use_failmalloc.patch: Patch to use pyfailmalloc on the Python test suite. Command to run test suite: "gdb -args ./python -m test -F".

call_exc_trace.patch: Fix for the call_exc_trace() function when the traceback is not set.

type_slot_tp_str.patch: Fix for the __str__ slot of types (slot_tp_str() in typeobject.c).

misc_fixes.patch: Misc bugfixes related to MemoryError, this patch should be read carefully. The changes in unicodeobject.c may be written differently. I wrote them to fix a failing assertion ("assert(data != NULL);") in _PyUnicode_CheckConsistency(), called when the creation of a new string failed.

New changeset 4f730c045f5f by Victor Stinner in branch 'default':
Issue bpo-18408: Fix call_exc_trace(): if the traceback is NULL, use None when
http://hg.python.org/cpython/rev/4f730c045f5f

New changeset 01a46dc00fc8 by Victor Stinner in branch 'default':
Issue bpo-18408: slot_tp_str() must not fallback on slot_tp_repr() on error
http://hg.python.org/cpython/rev/01a46dc00fc8

New changeset 39eb1ce5f377 by Victor Stinner in branch 'default':
Issue bpo-18408: ste_new() initialize all attributes before handling error
http://hg.python.org/cpython/rev/39eb1ce5f377

New changeset aaa6e8b8a5c9 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix compiler_import() to handle PyUnicode_Substring() failure properly
http://hg.python.org/cpython/rev/aaa6e8b8a5c9

New changeset ff93930a53c0 by Victor Stinner in branch 'default':
Issue bpo-18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/ff93930a53c0

New changeset 1a1869baec4c by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _Pickler_New() and _Unpickler_New(): initialize all
http://hg.python.org/cpython/rev/1a1869baec4c

New changeset e11121b9bd09 by Victor Stinner in branch 'default':
Issue bpo-18408: _elementtree.c now handles create_extra() failure
http://hg.python.org/cpython/rev/e11121b9bd09

New changeset 0877e17fa25e by Victor Stinner in branch 'default':
Issue bpo-18408: Different fixes in _elementtree.c to handle correctly MemoryError
http://hg.python.org/cpython/rev/0877e17fa25e

New changeset 51eddca9dd6f by Victor Stinner in branch 'default':
Issue bpo-18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/51eddca9dd6f

New changeset 5e50f1a0c985 by Victor Stinner in branch 'default':
Issue bpo-18408: In debug mode, PyCFunction_Call() now checks if an exception was
http://hg.python.org/cpython/rev/5e50f1a0c985

New changeset 4975bcd67aa7 by Victor Stinner in branch 'default':
Issue bpo-18408: normalizestring() now raises MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/4975bcd67aa7

New changeset 1eac89af9acf by Victor Stinner in branch 'default':
Issue bpo-18408: Oh, I was wrong: Pickler_New() must call Py_DECREF() to destroy
http://hg.python.org/cpython/rev/1eac89af9acf

New changeset d9446c2a2fd4 by Victor Stinner in branch 'default':
Issue bpo-18408: ceval.c: in debug mode, convert the PyErr_Occurred() check on
http://hg.python.org/cpython/rev/d9446c2a2fd4

New changeset 2f7c4df5cc46 by Victor Stinner in branch 'default':
Issue bpo-18408: errors.c: in debug mode, calling PyErr_BadInternalCall() now
http://hg.python.org/cpython/rev/2f7c4df5cc46

New changeset affb87b1d7ec by Victor Stinner in branch 'default':
Issue bpo-18408: pmerge() help of mro_implementation() now raises MemoryError on
http://hg.python.org/cpython/rev/affb87b1d7ec

New changeset f85fcbbbe8de by Victor Stinner in branch 'default':
Issue bpo-18408: _PyMemoTable_ResizeTable() now restores the old table if
http://hg.python.org/cpython/rev/f85fcbbbe8de

New changeset bb5da6e795a1 by Victor Stinner in branch 'default':
Issue bpo-18408: _pickle.c: Add missing PyErr_NoMemory() on memory allocation failures
http://hg.python.org/cpython/rev/bb5da6e795a1

New changeset 89c6495d1ff2 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
http://hg.python.org/cpython/rev/89c6495d1ff2

New changeset c80a9705803a by Victor Stinner in branch 'default':
Issue bpo-18408: parser module: fix error handling in node2tuple()
http://hg.python.org/cpython/rev/c80a9705803a

New changeset 60b1d7967ef8 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix create_extra() of _elementtree.c, raise MemoryError on memory
http://hg.python.org/cpython/rev/60b1d7967ef8

New changeset 1e0afd558ba3 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix constructors of _elementtree.c
http://hg.python.org/cpython/rev/1e0afd558ba3

listpop.patch: list_ass_slice() now handles listresize() failure when items are deleted.

frame_fasttolocals.patch: Add a new PyFrame_FastToLocalsWithError() function, same than PyFrame_FastToLocals(), except that errors are reported.

This function is not atomic: if an error occurs in the middle of map_to_dict(), the frame can enter an inconsistent state. Is it worse than not reporting the failure?

New changeset 735e3c43e61c by Victor Stinner in branch 'default':
Issue bpo-18408: Fix pyexpat.ParserCreate()
http://hg.python.org/cpython/rev/735e3c43e61c

New changeset 27f8ccce687d by Victor Stinner in branch 'default':
Issue bpo-18408: Fix CJK decoders, raise MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/27f8ccce687d

New changeset 82b2ee140994 by Victor Stinner in branch 'default':
Issue bpo-18408: PyObject_Call() now fails with an assertion error in debug mode
http://hg.python.org/cpython/rev/82b2ee140994

New changeset ef0dbe00760e by Victor Stinner in branch 'default':
Issue bpo-18408: Don't check unicode consistency in _PyUnicode_HAS_UTF8_MEMORY()
http://hg.python.org/cpython/rev/ef0dbe00760e

New changeset c676fc74d779 by Victor Stinner in branch 'default':
Issue bpo-18408: type_new() and PyType_FromSpecWithBases() now raise MemoryError
http://hg.python.org/cpython/rev/c676fc74d779

New changeset 9244e79c6e14 by Victor Stinner in branch 'default':
Issue bpo-18408: random_seed() now raises a MemoryError on memory allocation
http://hg.python.org/cpython/rev/9244e79c6e14

New changeset 71a572a516f9 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyEval_EvalFrameEx() for MemoryError
http://hg.python.org/cpython/rev/71a572a516f9

New changeset 2e42c0a39321 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _Py_DisplaySourceLine()
http://hg.python.org/cpython/rev/2e42c0a39321

New changeset 92a9ccb2a521 by Victor Stinner in branch 'default':
Issue bpo-18408: add more assertions on PyErr_Occurred() in ceval.c to detect bugs
http://hg.python.org/cpython/rev/92a9ccb2a521

New changeset 9213313b1a6f by Victor Stinner in branch 'default':
Issue bpo-18408: Fix show_warning(), clear also the exception raised by
http://hg.python.org/cpython/rev/9213313b1a6f

I created the issue bpo-18469: "PyStructSequence does not handle exceptions correctly".

import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure.

New issue bpo-18470: "tokenizer.c does not handle new_string() failure".

New changeset 533eb9ab895a by Victor Stinner in branch 'default':
Issue bpo-18408: Fix fileio_read() on _PyBytes_Resize() failure
http://hg.python.org/cpython/rev/533eb9ab895a

New changeset f0efd7ea1627 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix cjkcodecs decoders, add a new MBERR_EXCEPTION constant to
http://hg.python.org/cpython/rev/f0efd7ea1627

New changeset 395e67f348e7 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix list.extend(), handle list_resize() failure
http://hg.python.org/cpython/rev/395e67f348e7

New changeset ac4e8e6a7436 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyDict_GetItemString(), suppress PyUnicode_FromString() error
http://hg.python.org/cpython/rev/ac4e8e6a7436

New changeset b7c6d203c9fb by Victor Stinner in branch 'default':
Issue bpo-18408: dict_new() now fails on new_keys_object() error
http://hg.python.org/cpython/rev/b7c6d203c9fb

New changeset 28ff7ac91477 by Victor Stinner in branch 'default':
Issue bpo-18408: Py_ReprLeave() now saves/restores the current exception,
http://hg.python.org/cpython/rev/28ff7ac91477

New changeset 6bd01a59762a by Victor Stinner in branch 'default':
Issue bpo-18408: handle PySys_GetObject() failure, raise a RuntimeError
http://hg.python.org/cpython/rev/6bd01a59762a

You are great!

New changeset c8696ee49291 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix typo in build_node_tree() of the parser module
http://hg.python.org/cpython/rev/c8696ee49291

New changeset cf8f42eadbd0 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix Python-ast.c: handle init_types() failure (ex: MemoryError)
http://hg.python.org/cpython/rev/cf8f42eadbd0

New changeset a685f4c6e0b6 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyErr_NormalizeException(), handle PyObject_IsSubclass() failure
http://hg.python.org/cpython/rev/a685f4c6e0b6

New changeset 7fe4a0c0e905 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix locale.localeconv(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/7fe4a0c0e905

New changeset 8c1ca6720246 by Victor Stinner in branch 'default':
Issue bpo-18408: Handle PyArena_AddPyObject() failure in ast.c
http://hg.python.org/cpython/rev/8c1ca6720246

New changeset 1872555a2f4e by Victor Stinner in branch 'default':
Issue bpo-18408: Mention changes in Misc/NEWS
http://hg.python.org/cpython/rev/1872555a2f4e

@serhiy: Would like to review attached patches listpop.patch (msg193130) and frame_fasttolocals.patch (msg193131)? (Other are already commited)

Oops.

@serhiy: Would *you* like to review attached patches ...

Another buildbot failure:

[150/375/4] test_ctypes
Assertion failed: __EX, file Python/ceval.c, line 4064
Fatal Python error: Aborted

Current thread 0x00000001:
File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 44 in capture_stderr
File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 63 in test_FloatDivisionError

http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/310/steps/test/logs/stdio

This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree:

    def test_parser_target_error_in_start(self):
        assertEqual = self.assertEqual

        events = []
        class Target(object):
            def start(self, tag, attrib):
                events.append("start")
                assertEqual("TAG", tag)
                raise ValueError("TEST")
            def end(self, tag):
                events.append("end")
                assertEqual("TAG", tag)
            def close(self):
                return "DONE"

        parser = ET.XMLParser(target=Target())

    try:
        parser.feed("<TAG/>")
    except ValueError:
        self.assertTrue('TEST' in str(sys.exc_info()[1]))
    else:
        self.assertTrue(False)
    self.assertTrue("start" in events)

Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.

(note: this is not the exact code, I removed a couple of conditions that are not used by ET)

New changeset 6ec0e9347dd4 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _elementtree.c, don't call Python function from an expat
http://hg.python.org/cpython/rev/6ec0e9347dd4

This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree: (...)

Thanks for the quick report! Are you testing automatically lxml with the Python HEAD?

The issue should be fixed by the changeset 6ec0e9347dd4. I also opened the issue bpo-18501 to discuss how ElementTree should handle Python exceptions.

New changeset 5a6cdc0d7de1 by Victor Stinner in branch 'default':
Issue bpo-18501, bpo-18408: Fix expat handlers in pyexpat, don't call Python
http://hg.python.org/cpython/rev/5a6cdc0d7de1

I created the issue bpo-18488: "sqlite: finalize() method of user function may be called with an exception set if a call to step() method failed".

@victor: yes, the Cython project has CI tests running against debug builds of all CPython branches since 2.4, updated daily. lxml is part of an extended set of tests for Cython, and the test suite of lxml includes several compatibility tests for ElementTree.

https://sage.math.washington.edu:8091/hudson/

https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk/

I noticed the failures when those tests started failing. Thanks for the quick fix, and actually for all the work you're doing here.

New changeset 3f25a7dd8346 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix list_ass_slice(), handle list_resize() failure
http://hg.python.org/cpython/rev/3f25a7dd8346

"import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure."

I created the issue bpo-18507 for that.

"fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION"

I created the issue bpo-18509 for that.

New changeset 026593cbc006 by Victor Stinner in branch 'default':
Issue bpo-18519, bpo-18408: Fix sqlite authorizer callback
http://hg.python.org/cpython/rev/026593cbc006

"""
python: Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted

Current thread 0x404df8c0:
File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 441 in authorizer_cb
"""

Ned Deily created the issue bpo-18519 for that. It should be fixed by the changeset 026593cbc006.

New changeset de1ea7f84e08 by Victor Stinner in branch 'default':
Issue bpo-18609, bpo-18408: _ctypes_add_traceback() now clears the current exception
http://hg.python.org/cpython/rev/de1ea7f84e08

"""
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/98/steps/test/logs/stdio

FAIL: test_finalize_runnning_thread (test.test_threading.ThreadTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threading.py", line 298, in test_finalize_runnning_thread
    self.assertEqual(rc, 42)
AssertionError: -6 != 42
"""

Oh, this error is not a regression introduced by this issue: it was already present in the oldest build of this buildbot:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/5/steps/test/logs/stdio

TODO:

• review and integrate frame_fasttolocals.patch
• bpo-18507: import_init() should not use Py_FatalError() but return an error
• bpo-18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error
• fix tests hang when an exception occurs in a thread

Most bugs have been fixed.

New changeset 42aecc601a41 by Antoine Pitrou in branch 'default':
Back out 5bd9db528aed (issue bpo-18408). It caused unsolved buildbot failures.
http://hg.python.org/cpython/rev/42aecc601a41

Victor, I'll let you find out how to improve your debug checks, but in the meantime I've reverted the commit that randomly broke some Windows buildbots.

I didn't know that buildbots are failing. Do you have at least an example?
An error message, anything?

It's reported in bpo-18664 (you're nosied there :-)).

New changeset 79ce25c70795 by Victor Stinner in branch 'default':
Issue bpo-18408: _PyObject_Dump() now saves/restores the current exception
http://hg.python.org/cpython/rev/79ce25c70795

New changeset e63f19d0a651 by Victor Stinner in branch 'default':
Issue bpo-18664, bpo-18408: Rewrite PyErr_WriteUnraisable() to handle errors
http://hg.python.org/cpython/rev/e63f19d0a651

New changeset 8fb3a6f9b0a4 by Victor Stinner in branch 'default':
Restore changeset 5bd9db528aed (issue bpo-18408)
http://hg.python.org/cpython/rev/8fb3a6f9b0a4

New changeset 4ef4578db38a by Victor Stinner in branch 'default':
Issue bpo-18408: Add a new PyFrame_FastToLocalsWithError() function to handle
http://hg.python.org/cpython/rev/4ef4578db38a

Ok, most critical bugs have been fixed. If I find new bugs, I will open new issues.

New changeset e1d51c42e5a1 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
http://hg.python.org/cpython/rev/e1d51c42e5a1

New changeset 7f4a976829f1 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix zipimport, handle PyUnicode_Substring() and get_subname() failures
http://hg.python.org/cpython/rev/7f4a976829f1

New changeset 4749c3ac0654 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix iobase_readline(), handle PyByteArray_Resize() failure
http://hg.python.org/cpython/rev/4749c3ac0654

New changeset 549cd815bf8d by Victor Stinner in branch 'default':
Issue bpo-18408: Fix error handling in PyBytes_FromObject()
http://hg.python.org/cpython/rev/549cd815bf8d

New changeset 9d9371d4ff7b by Victor Stinner in branch 'default':
Issue bpo-18408: Fix bytearrayiter.partition()/rpartition(), handle
http://hg.python.org/cpython/rev/9d9371d4ff7b

New changeset 20cd15a28ad3 by Victor Stinner in branch 'default':
Issue bpo-18408: Fix _ctypes_alloc_format_string(), raise MemoryError on memory
http://hg.python.org/cpython/rev/20cd15a28ad3

New changeset 1386fb31c0ed by Victor Stinner in branch 'default':
Issue bpo-18408: Fix PyCStructUnionType_update_stgdict(), handle
http://hg.python.org/cpython/rev/1386fb31c0ed

New changeset 8f556ee0f6ba by Antoine Pitrou in branch 'default':
Fix refleak introduced by 4f730c045f5f (issue bpo-18408) and unveiled by 95eea8624d05 (issue bpo-16596).
http://hg.python.org/cpython/rev/8f556ee0f6ba

New changeset b742c1c5c0bf by Victor Stinner in branch 'default':
PEP-490: add bpo-18408
https://hg.python.org/peps/rev/b742c1c5c0bf