classification
Title: Fixes crashes found by pyfailmalloc
Type: crash Stage:
Components: Versions: Python 3.4
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: haypo Nosy List: gregory.p.smith, haypo, neologix, pitrou, python-dev, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2013-07-08 20:10 by haypo, last changed 2013-11-23 13:13 by python-dev. This issue is now closed.

Files
File name Uploaded Description Edit
use_failmalloc.patch haypo, 2013-07-08 23:59 review
call_exc_trace.patch haypo, 2013-07-09 00:00 review
type_slot_tp_str.patch haypo, 2013-07-09 00:01 review
misc_fixes.patch haypo, 2013-07-09 00:03 review
listpop.patch haypo, 2013-07-15 22:06
frame_fasttolocals.patch haypo, 2013-07-15 22:08 review
Messages (68)
msg192691 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-08 20:10
I created a first toy based on the PEP 445 to test of Python handles MemoryError: in short, Python handles them badly...

Home page of the project:
https://pypi.python.org/pypi/pyfailmalloc

I will use this issue to track crashes found by this tool.
msg192693 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-08 20:36
New changeset 111c2a070f28 by Victor Stinner in branch 'default':
Issue #18408: PyObject_GC_NewVar() now raises SystemError exception if nitems
http://hg.python.org/cpython/rev/111c2a070f28

New changeset ba766323b53a by Victor Stinner in branch 'default':
Issue #18408: Fix PyDict_New() to handle correctly new_keys_object() failure
http://hg.python.org/cpython/rev/ba766323b53a

New changeset 68887e177dd4 by Victor Stinner in branch 'default':
Issue #18408: Fix list.pop() to handle list_resize() failure (MemoryError).
http://hg.python.org/cpython/rev/68887e177dd4

New changeset 697d722d97f9 by Victor Stinner in branch 'default':
Issue #18408: Fix marshal reader for Unicode strings: handle
http://hg.python.org/cpython/rev/697d722d97f9

New changeset de1473f4503b by Victor Stinner in branch 'default':
Issue #18408: Fix PyType_Ready() and type.__bases__ setter to handle
http://hg.python.org/cpython/rev/de1473f4503b

New changeset f4311870e329 by Victor Stinner in branch 'default':
Issue #18408: Fix call_function() of ceval.c to handle PyTuple_New() failure
http://hg.python.org/cpython/rev/f4311870e329

New changeset df8b40593a08 by Victor Stinner in branch 'default':
Issue #18408: Fix _PyUnicodeWriter_Finish(): clear writer->buffer,
http://hg.python.org/cpython/rev/df8b40593a08
msg192694 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-08 20:50
"I created a first toy based on the PEP 445 to test of Python handles MemoryError: in short, Python handles them badly..."

Ooops: to test *how* Python handles..
msg192701 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-08 22:53
New changeset ba79f6a86300 by Victor Stinner in branch 'default':
Issue #18408: Fix ConvParam() of the ctypes module to handle paramfunc failure
http://hg.python.org/cpython/rev/ba79f6a86300

New changeset 2ef2edfd1a4c by Victor Stinner in branch 'default':
Issue #18408: Fix zlib.compressobj() to handle PyThread_allocate_lock() failure
http://hg.python.org/cpython/rev/2ef2edfd1a4c

New changeset a45407fa4a5b by Victor Stinner in branch 'default':
Issue #18408: Fix PyCode_Optimize(): raise a MemoryError on memory allocation
http://hg.python.org/cpython/rev/a45407fa4a5b

New changeset ed0c9d77e179 by Victor Stinner in branch 'default':
Issue #18408: Fix usage of _PyBytes_Resize()
http://hg.python.org/cpython/rev/ed0c9d77e179

New changeset d605c6b8095c by Victor Stinner in branch 'default':
Issue #18408: _PyUnicodeWriter_Finish() now clears its buffer attribute in all
http://hg.python.org/cpython/rev/d605c6b8095c

New changeset 549d8d3297f2 by Victor Stinner in branch 'default':
Issue #18408: Fix _PyMem_DebugRealloc()
http://hg.python.org/cpython/rev/549d8d3297f2

New changeset c91e7f707562 by Victor Stinner in branch 'default':
Issue #18408: Fix select.select() to handle PyList_New() failure (MemoryError)
http://hg.python.org/cpython/rev/c91e7f707562
msg192703 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-08 23:59
use_failmalloc.patch: Patch to use pyfailmalloc on the Python test suite. Command to run test suite: "gdb -args ./python -m test -F".
msg192704 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-09 00:00
call_exc_trace.patch: Fix for the call_exc_trace() function when the traceback is not set.
msg192705 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-09 00:01
type_slot_tp_str.patch: Fix for the __str__ slot of types (slot_tp_str() in typeobject.c).
msg192706 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-09 00:03
misc_fixes.patch: Misc bugfixes related to MemoryError, this patch should be read carefully. The changes in unicodeobject.c may be written differently. I wrote them to fix a failing assertion ("assert(data != NULL);") in _PyUnicode_CheckConsistency(), called when the creation of a new string failed.
msg192797 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-10 12:04
New changeset 4f730c045f5f by Victor Stinner in branch 'default':
Issue #18408: Fix call_exc_trace(): if the traceback is NULL, use None when
http://hg.python.org/cpython/rev/4f730c045f5f
msg192902 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-11 20:46
New changeset 01a46dc00fc8 by Victor Stinner in branch 'default':
Issue #18408: slot_tp_str() must not fallback on slot_tp_repr() on error
http://hg.python.org/cpython/rev/01a46dc00fc8
msg192908 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-11 21:12
New changeset 39eb1ce5f377 by Victor Stinner in branch 'default':
Issue #18408: ste_new() initialize all attributes before handling error
http://hg.python.org/cpython/rev/39eb1ce5f377

New changeset aaa6e8b8a5c9 by Victor Stinner in branch 'default':
Issue #18408: Fix compiler_import() to handle PyUnicode_Substring() failure properly
http://hg.python.org/cpython/rev/aaa6e8b8a5c9

New changeset ff93930a53c0 by Victor Stinner in branch 'default':
Issue #18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/ff93930a53c0

New changeset 1a1869baec4c by Victor Stinner in branch 'default':
Issue #18408: Fix _Pickler_New() and _Unpickler_New(): initialize all
http://hg.python.org/cpython/rev/1a1869baec4c

New changeset e11121b9bd09 by Victor Stinner in branch 'default':
Issue #18408: _elementtree.c now handles create_extra() failure
http://hg.python.org/cpython/rev/e11121b9bd09

New changeset 0877e17fa25e by Victor Stinner in branch 'default':
Issue #18408: Different fixes in _elementtree.c to handle correctly MemoryError
http://hg.python.org/cpython/rev/0877e17fa25e
msg192913 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-11 21:47
New changeset 51eddca9dd6f by Victor Stinner in branch 'default':
Issue #18408: parsetok() must not write into stderr on memory allocation error
http://hg.python.org/cpython/rev/51eddca9dd6f

New changeset 5e50f1a0c985 by Victor Stinner in branch 'default':
Issue #18408: In debug mode, PyCFunction_Call() now checks if an exception was
http://hg.python.org/cpython/rev/5e50f1a0c985
msg192917 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-11 22:59
New changeset 4975bcd67aa7 by Victor Stinner in branch 'default':
Issue #18408: normalizestring() now raises MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/4975bcd67aa7

New changeset 1eac89af9acf by Victor Stinner in branch 'default':
Issue #18408: Oh, I was wrong: Pickler_New() must call Py_DECREF() to destroy
http://hg.python.org/cpython/rev/1eac89af9acf

New changeset d9446c2a2fd4 by Victor Stinner in branch 'default':
Issue #18408: ceval.c: in debug mode, convert the PyErr_Occurred() check on
http://hg.python.org/cpython/rev/d9446c2a2fd4

New changeset 2f7c4df5cc46 by Victor Stinner in branch 'default':
Issue #18408: errors.c: in debug mode, calling PyErr_BadInternalCall() now
http://hg.python.org/cpython/rev/2f7c4df5cc46

New changeset affb87b1d7ec by Victor Stinner in branch 'default':
Issue #18408: pmerge() help of mro_implementation() now raises MemoryError on
http://hg.python.org/cpython/rev/affb87b1d7ec

New changeset f85fcbbbe8de by Victor Stinner in branch 'default':
Issue #18408: _PyMemoTable_ResizeTable() now restores the old table if
http://hg.python.org/cpython/rev/f85fcbbbe8de

New changeset bb5da6e795a1 by Victor Stinner in branch 'default':
Issue #18408: _pickle.c: Add missing PyErr_NoMemory() on memory allocation failures
http://hg.python.org/cpython/rev/bb5da6e795a1
msg192918 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-11 23:42
New changeset 89c6495d1ff2 by Victor Stinner in branch 'default':
Issue #18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
http://hg.python.org/cpython/rev/89c6495d1ff2

New changeset c80a9705803a by Victor Stinner in branch 'default':
Issue #18408: parser module: fix error handling in node2tuple()
http://hg.python.org/cpython/rev/c80a9705803a
msg192920 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-12 00:07
New changeset 60b1d7967ef8 by Victor Stinner in branch 'default':
Issue #18408: Fix create_extra() of _elementtree.c, raise MemoryError on memory
http://hg.python.org/cpython/rev/60b1d7967ef8

New changeset 1e0afd558ba3 by Victor Stinner in branch 'default':
Issue #18408: Fix constructors of _elementtree.c
http://hg.python.org/cpython/rev/1e0afd558ba3
msg193130 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-15 22:06
listpop.patch: list_ass_slice() now handles listresize() failure when items are deleted.
msg193131 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-15 22:08
frame_fasttolocals.patch: Add a new PyFrame_FastToLocalsWithError() function, same than PyFrame_FastToLocals(), except that errors are reported.

This function is not atomic: if an error occurs in the middle of map_to_dict(), the frame can enter an inconsistent state. Is it worse than not reporting the failure?
msg193132 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-15 22:12
New changeset 735e3c43e61c by Victor Stinner in branch 'default':
Issue #18408: Fix pyexpat.ParserCreate()
http://hg.python.org/cpython/rev/735e3c43e61c

New changeset 27f8ccce687d by Victor Stinner in branch 'default':
Issue #18408: Fix CJK decoders, raise MemoryError on memory allocation failure
http://hg.python.org/cpython/rev/27f8ccce687d

New changeset 82b2ee140994 by Victor Stinner in branch 'default':
Issue #18408: PyObject_Call() now fails with an assertion error in debug mode
http://hg.python.org/cpython/rev/82b2ee140994

New changeset ef0dbe00760e by Victor Stinner in branch 'default':
Issue #18408: Don't check unicode consistency in _PyUnicode_HAS_UTF8_MEMORY()
http://hg.python.org/cpython/rev/ef0dbe00760e

New changeset c676fc74d779 by Victor Stinner in branch 'default':
Issue #18408: type_new() and PyType_FromSpecWithBases() now raise MemoryError
http://hg.python.org/cpython/rev/c676fc74d779

New changeset 9244e79c6e14 by Victor Stinner in branch 'default':
Issue #18408: random_seed() now raises a MemoryError on memory allocation
http://hg.python.org/cpython/rev/9244e79c6e14

New changeset 71a572a516f9 by Victor Stinner in branch 'default':
Issue #18408: Fix PyEval_EvalFrameEx() for MemoryError
http://hg.python.org/cpython/rev/71a572a516f9
msg193138 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-16 00:02
New changeset 2e42c0a39321 by Victor Stinner in branch 'default':
Issue #18408: Fix _Py_DisplaySourceLine()
http://hg.python.org/cpython/rev/2e42c0a39321

New changeset 92a9ccb2a521 by Victor Stinner in branch 'default':
Issue #18408: add more assertions on PyErr_Occurred() in ceval.c to detect bugs
http://hg.python.org/cpython/rev/92a9ccb2a521

New changeset 9213313b1a6f by Victor Stinner in branch 'default':
Issue #18408: Fix show_warning(), clear also the exception raised by
http://hg.python.org/cpython/rev/9213313b1a6f
msg193140 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-16 00:33
I created the issue #18469: "PyStructSequence does not handle exceptions correctly".
msg193142 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-16 01:28
import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure.
msg193144 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-16 01:32
New issue #18470: "tokenizer.c does not handle new_string() failure".
msg193192 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-16 21:09
New changeset 533eb9ab895a by Victor Stinner in branch 'default':
Issue #18408: Fix fileio_read() on _PyBytes_Resize() failure
http://hg.python.org/cpython/rev/533eb9ab895a

New changeset f0efd7ea1627 by Victor Stinner in branch 'default':
Issue #18408: Fix cjkcodecs decoders, add a new MBERR_EXCEPTION constant to
http://hg.python.org/cpython/rev/f0efd7ea1627

New changeset 395e67f348e7 by Victor Stinner in branch 'default':
Issue #18408: Fix list.extend(), handle list_resize() failure
http://hg.python.org/cpython/rev/395e67f348e7

New changeset ac4e8e6a7436 by Victor Stinner in branch 'default':
Issue #18408: Fix PyDict_GetItemString(), suppress PyUnicode_FromString() error
http://hg.python.org/cpython/rev/ac4e8e6a7436

New changeset b7c6d203c9fb by Victor Stinner in branch 'default':
Issue #18408: dict_new() now fails on new_keys_object() error
http://hg.python.org/cpython/rev/b7c6d203c9fb

New changeset 28ff7ac91477 by Victor Stinner in branch 'default':
Issue #18408: Py_ReprLeave() now saves/restores the current exception,
http://hg.python.org/cpython/rev/28ff7ac91477

New changeset 6bd01a59762a by Victor Stinner in branch 'default':
Issue #18408: handle PySys_GetObject() failure, raise a RuntimeError
http://hg.python.org/cpython/rev/6bd01a59762a
msg193196 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-16 22:28
You are great!
msg193198 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-16 23:01
New changeset c8696ee49291 by Victor Stinner in branch 'default':
Issue #18408: Fix typo in build_node_tree() of the parser module
http://hg.python.org/cpython/rev/c8696ee49291

New changeset cf8f42eadbd0 by Victor Stinner in branch 'default':
Issue #18408: Fix Python-ast.c: handle init_types() failure (ex: MemoryError)
http://hg.python.org/cpython/rev/cf8f42eadbd0

New changeset a685f4c6e0b6 by Victor Stinner in branch 'default':
Issue #18408: Fix PyErr_NormalizeException(), handle PyObject_IsSubclass() failure
http://hg.python.org/cpython/rev/a685f4c6e0b6

New changeset 7fe4a0c0e905 by Victor Stinner in branch 'default':
Issue #18408: Fix locale.localeconv(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/7fe4a0c0e905

New changeset 8c1ca6720246 by Victor Stinner in branch 'default':
Issue #18408: Handle PyArena_AddPyObject() failure in ast.c
http://hg.python.org/cpython/rev/8c1ca6720246
msg193199 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-16 23:10
New changeset 1872555a2f4e by Victor Stinner in branch 'default':
Issue #18408: Mention changes in Misc/NEWS
http://hg.python.org/cpython/rev/1872555a2f4e
msg193200 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-16 23:12
@Serhiy: Would like to review attached patches listpop.patch (msg193130) and frame_fasttolocals.patch (msg193131)? (Other are already commited)
msg193202 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-16 23:24
Oops.

@Serhiy: Would *you* like to review attached patches ...
msg193209 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-17 07:33
listpop.patch:

list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.
msg193211 - (view) Author: Charles-Fran├žois Natali (neologix) * (Python committer) Date: 2013-07-17 08:47
Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).
msg193213 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-07-17 08:51
> Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?

Seconded.
msg193218 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-17 10:12
2013/7/17 Serhiy Storchaka <report@bugs.python.org>:
> listpop.patch:
>
> list_resize(a, Py_SIZE(a) + d) for d < 0 fails only when PyMem_Realloc(p, n) returns NULL if n < allocated size. Is it possible? Is it desired behavior? Perhaps we should declare that PyMem_Realloc() should return the original pointer if it can't shrink an allocated block.

Resizing a buffer to make it smaller "should" not fail, and if it
fails, the case is very unlikely. But I prefer to handle the case,
rather than hoping that the case cannot occur in the libc
implementation.

For example, _PyObject_Realloc() of pymalloc allocates a *new* buffer
to shrink an existing buffer if the size is reduced by more than 25%:

        if (nbytes <= size) {
            /* The block is staying the same or shrinking.  If
             * it's shrinking, there's a tradeoff:  it costs
             * cycles to copy the block to a smaller size class,
             * but it wastes memory not to copy it.  The
             * compromise here is to copy on shrink only if at
             * least 25% of size can be shaved off.
             */
            if (4 * nbytes > 3 * size) {
                /* It's the same,
                 * or shrinking and new/old > 3/4.
                 */
                return p;
            }
            size = nbytes;
        }
        bp = _PyObject_Malloc(ctx, nbytes);

I plan also to test Python with PyMem_Malloc using pymalloc to check
if it's faster or not. So PyMem_Realloc() may fail when shrinking a
buffer ;-)

By the way, I already fixed similar bugs (handle PyMem_Realloc failure
when shrinking the buffer) in this issue. Examples:
http://hg.python.org/cpython/rev/68887e177dd4
http://hg.python.org/cpython/rev/549d8d3297f2

listpop.patch only changes the error handling to make it more
resilient when the "very unlikely" case occurs ;-)
msg193219 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-17 10:24
2013/7/17 Charles-Fran├žois Natali <report@bugs.python.org>:
> Victor, how about adding pyfailmalloc to the main repo (maybe under Tools), with a script making it easy to run the tests suite with it enabled?
> This way, it'll make it easier to run it from time to time (one could eve imagine making it part of the test suite at some point, but that's another story...).

I plan to write a new module and propose it for inclusion in Python
3.4. It will have the features of pytracemalloc *and* pyfailmalloc,
and also statistics (number of calls to each allocator function, with
a function to reset counters).

The blocker pointer is to get the size of a buffer on free(). In
pytracemalloc, I'm using a glib hash lib: pointer => structure {size
of the buffer, filename, line number}. The dependency to glib should
be removed to integrate the module in Python :-)

The simpler solution is to allocate a wider buffer and store the
metadata directly in the buffer before the data. The second problem is
how this feature would be enabled. Enabling the feature at runtime
would require a new environment variable to set the right memory
allocator at startup. The choice of the memory allocator must be done
before the first memory allocation, so in main().

Integrating pytracemalloc into CPython was my main motivation to write
the PEP 445.

If you would like to use pyfailmalloc right now, you have to:

* build pyfailmalloc with Python 3.4
* apply debug_cpython.patch on Python which is the pyfailmalloc repository
* run gdb -x debug_cpython.gdb (which can also be found in the
pyfailmalloc repository)
* https://bitbucket.org/haypo/pyfailmalloc
msg193224 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-17 11:43
New changeset 7b81a535ad14 by Victor Stinner in branch 'default':
Issue #18408: Fix structseq_reduce(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/7b81a535ad14
msg193225 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-17 11:45
Open issues (TODO list):

- review and apply listpop.patch and frame_fasttolocals.patch
- fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION
- fix Modules/pyexpat.c: handlers do no detect failures
- fix sqlite: Modules/_sqlite/connection.c calls the finalize method on the aggregate whereas calling step() failed
- fix tests hang when an exception occurs in a thread
- fix new bugs found by pyfailmalloc :-)
msg193245 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-17 20:11
New changeset 4d3bb9763bf9 by Victor Stinner in branch 'default':
Issue #18408: Fix time.tzset(), detect exception when calling PyInit_timezone()
http://hg.python.org/cpython/rev/4d3bb9763bf9

New changeset 563b27bef79f by Victor Stinner in branch 'default':
Issue #18408: Fix heapq.heappop(), handle PyList_SetSlice() failure
http://hg.python.org/cpython/rev/563b27bef79f

New changeset 6fb3414710ab by Victor Stinner in branch 'default':
Issue #18408: Fix ast_for_atom(), PyErr_Fetch(&type, &value, &tback) can set value to NULL
http://hg.python.org/cpython/rev/6fb3414710ab

New changeset 479af6ca3a3c by Victor Stinner in branch 'default':
Issue #18408: Fix _PyImport_LoadDynamicModule(), handle PyUnicode_FromFormat() failure
http://hg.python.org/cpython/rev/479af6ca3a3c

New changeset 573f7d485ce8 by Victor Stinner in branch 'default':
Issue #18408: Fix PyErr_SetImportError(), handle PyDict_SetItemString() failure
http://hg.python.org/cpython/rev/573f7d485ce8

New changeset 97bb3bdf1443 by Victor Stinner in branch 'default':
Issue #18408: Fix listpop(), handle list_ass_slice() failure
http://hg.python.org/cpython/rev/97bb3bdf1443

New changeset 2f5f1db8eb88 by Victor Stinner in branch 'default':
Issue #18408: Fix Py_ReprEnter(), handle PyList_Append() failure
http://hg.python.org/cpython/rev/2f5f1db8eb88

New changeset c2d90ff0780c by Victor Stinner in branch 'default':
Issue #18408: Fix PyType_Ready(), handle _PyDict_SetItemId() failure
http://hg.python.org/cpython/rev/c2d90ff0780c
msg193250 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-17 20:33
New changeset 736514fb7b52 by Victor Stinner in branch 'default':
Issue #18408: Rewrite NEGATE() macro in longobject.c to handle PyLong_FromLong() failure
http://hg.python.org/cpython/rev/736514fb7b52
msg193258 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-17 23:46
New changeset d85b0c355052 by Victor Stinner in branch 'default':
Issue #18408: Fix array_index(), handle getarrayitem() failure
http://hg.python.org/cpython/rev/d85b0c355052

New changeset b05a6a6eb525 by Victor Stinner in branch 'default':
Issue #18408: Fix array_tolist(), handle PyList_SetItem() failure
http://hg.python.org/cpython/rev/b05a6a6eb525

New changeset 161de66cae49 by Victor Stinner in branch 'default':
Issue #18408: Fix dict_repr(), don't call PyObject_Repr() with an exception set
http://hg.python.org/cpython/rev/161de66cae49

New changeset 48a869a39e2d by Victor Stinner in branch 'default':
Issue #18408: PyEval_EvalFrameEx() and PyEval_CallObjectWithKeywords() now fail
http://hg.python.org/cpython/rev/48a869a39e2d
msg193261 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-17 23:57
New changeset 9bae7696951f by Victor Stinner in branch 'default':
Issue #18408: Fix _pysqlite_fetch_one_row(), in debug mode, don't call
http://hg.python.org/cpython/rev/9bae7696951f

New changeset 5bd9db528aed by Victor Stinner in branch 'default':
Issue #18408: PyObject_Str(), PyObject_Repr() and type_call() now fail with an
http://hg.python.org/cpython/rev/5bd9db528aed
msg193262 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-18 00:31
New changeset ddff866d820d by Victor Stinner in branch 'default':
Issue #18408: Fix PyInit__curses_panel(), handle import_curses() failure
http://hg.python.org/cpython/rev/ddff866d820d
msg193263 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-18 00:51
Changesets 48a869a39e2d and 5bd9db528aed adding "assert(!PyErr_Occurred());" broke some buildbots. I fixed the issue on OpenSolaris, I'm unable to reproduce the 2 remaining issues (I tested on Fedora 18 and FreeBSD 9).



http://buildbot.python.org/all/builders/x86%20Ubuntu%20Shared%203.x/builds/8236/steps/test/logs/stdio

python: Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted

Current thread 0x404df8c0:
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 441 in authorizer_cb
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 406 in test_table_access
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/unittest/case.py", line 496 in run
  ...


http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/98/steps/test/logs/stdio

FAIL: test_finalize_runnning_thread (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threading.py", line 298, in test_finalize_runnning_thread
    self.assertEqual(rc, 42)
AssertionError: -6 != 42
msg193264 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-18 01:08
Another buildbot failure:


[150/375/4] test_ctypes
Assertion failed: __EX, file  Python/ceval.c, line 4064
Fatal Python error: Aborted

Current thread 0x00000001:
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 44 in capture_stderr
  File "/home/shager/cpython-buildarea/3.x.edelsohn-aix-ppc64/build/Lib/ctypes/test/test_random_things.py", line 63 in test_FloatDivisionError
msg193265 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-18 01:08
http://buildbot.python.org/all/builders/PPC64%20AIX%203.x/builds/310/steps/test/logs/stdio
msg193316 - (view) Author: Stefan Behnel (scoder) * Date: 2013-07-18 19:49
This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree:

    def test_parser_target_error_in_start(self):
        assertEqual = self.assertEqual

        events = []
        class Target(object):
            def start(self, tag, attrib):
                events.append("start")
                assertEqual("TAG", tag)
                raise ValueError("TEST")
            def end(self, tag):
                events.append("end")
                assertEqual("TAG", tag)
            def close(self):
                return "DONE"

        parser = ET.XMLParser(target=Target())

        try:
            parser.feed("<TAG/>")
        except ValueError:
            self.assertTrue('TEST' in str(sys.exc_info()[1]))
        else:
            self.assertTrue(False)
        self.assertTrue("start" in events)

Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.

(note: this is not the exact code, I removed a couple of conditions that are not used by ET)
msg193324 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-18 20:46
New changeset 6ec0e9347dd4 by Victor Stinner in branch 'default':
Issue #18408: Fix _elementtree.c, don't call Python function from an expat
http://hg.python.org/cpython/rev/6ec0e9347dd4
msg193327 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-18 21:07
> This test from lxml's ElementTree test suite crashes for me now when run against (c)ElementTree: (...)

Thanks for the quick report! Are you testing automatically lxml with the Python HEAD?

The issue should be fixed by the changeset 6ec0e9347dd4. I also opened the issue #18501 to discuss how ElementTree should handle Python exceptions.
msg193329 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-18 21:43
New changeset 5a6cdc0d7de1 by Victor Stinner in branch 'default':
Issue #18501, #18408: Fix expat handlers in pyexpat, don't call Python
http://hg.python.org/cpython/rev/5a6cdc0d7de1
msg193337 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-18 23:43
I created the issue #18488: "sqlite: finalize() method of user function may be called with an exception set if a call to step() method failed".
msg193350 - (view) Author: Stefan Behnel (scoder) * Date: 2013-07-19 09:11
@Victor: yes, the Cython project has CI tests running against debug builds of all CPython branches since 2.4, updated daily. lxml is part of an extended set of tests for Cython, and the test suite of lxml includes several compatibility tests for ElementTree.

https://sage.math.washington.edu:8091/hudson/

https://sage.math.washington.edu:8091/hudson/job/cython-devel-lxml-trunk/

I noticed the failures when those tests started failing. Thanks for the quick fix, and actually for all the work you're doing here.
msg193374 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-19 21:14
New changeset 3f25a7dd8346 by Victor Stinner in branch 'default':
Issue #18408: Fix list_ass_slice(), handle list_resize() failure
http://hg.python.org/cpython/rev/3f25a7dd8346
msg193378 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-19 21:39
"import_init() calls Py_FatalError() at any error. This is not kind for Py_NewInterpreter() which calls import_init(): Py_NewInterpreter() exits Python with a fatal error, instead of returning NULL, on import_init() failure."

I created the issue #18507 for that.
msg193382 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-19 22:01
"fix CJK codecs: using UNIINV to report _PyUnicodeWriter_WriteChar() failure in Modules/cjkcodecs/cjkcodecs.h does not work. The caller is not aware of the failure: multibytecodec_decerror() receives e=1, whereas it should get MBERR_EXCEPTION"

I created the issue #18509 for that.
msg193435 - (view) Author: Roundup Robot (python-dev) Date: 2013-07-21 11:09
New changeset 026593cbc006 by Victor Stinner in branch 'default':
Issue #18519, #18408: Fix sqlite authorizer callback
http://hg.python.org/cpython/rev/026593cbc006
msg193438 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-07-21 11:15
"""
python: Python/ceval.c:1210: PyEval_EvalFrameEx: Assertion `!PyErr_Occurred()' failed.
Fatal Python error: Aborted

Current thread 0x404df8c0:
  File "/srv/buildbot/buildarea/3.x.bolen-ubuntu/build/Lib/sqlite3/test/userfunctions.py", line 441 in authorizer_cb
"""

Ned Deily created the issue #18519 for that. It should be fixed by the changeset 026593cbc006.
msg194129 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-01 22:49
New changeset de1ea7f84e08 by Victor Stinner in branch 'default':
Issue #18609, #18408: _ctypes_add_traceback() now clears the current exception
http://hg.python.org/cpython/rev/de1ea7f84e08
msg195112 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-08-14 00:11
"""
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/98/steps/test/logs/stdio

FAIL: test_finalize_runnning_thread (test.test_threading.ThreadTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/home/buildbot/koobs-freebsd10/3.x.koobs-freebsd10/build/Lib/test/test_threading.py", line 298, in test_finalize_runnning_thread
    self.assertEqual(rc, 42)
AssertionError: -6 != 42
"""

Oh, this error is not a regression introduced by this issue: it was already present in the oldest build of this buildbot:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%2010.0%203.x/builds/5/steps/test/logs/stdio
msg195113 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-08-14 00:19
TODO:

- review and integrate frame_fasttolocals.patch
- #18507: import_init() should not use Py_FatalError() but return an error
- #18509: CJK decoders should return MBERR_EXCEPTION on PyUnicodeWriter error
- fix tests hang when an exception occurs in a thread

Most bugs have been fixed.
msg196029 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-23 21:18
New changeset 42aecc601a41 by Antoine Pitrou in branch 'default':
Back out 5bd9db528aed (issue #18408).  It caused unsolved buildbot failures.
http://hg.python.org/cpython/rev/42aecc601a41
msg196030 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-23 21:19
Victor, I'll let you find out how to improve your debug checks, but in the meantime I've reverted the commit that randomly broke some Windows buildbots.
msg196041 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-08-23 21:54
I didn't know that buildbots are failing. Do you have at least an example?
An error message, anything?
msg196042 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-08-23 21:55
It's reported in issue18664 (you're nosied there :-)).
msg196199 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-26 12:06
New changeset 79ce25c70795 by Victor Stinner in branch 'default':
Issue #18408: _PyObject_Dump() now saves/restores the current exception
http://hg.python.org/cpython/rev/79ce25c70795

New changeset e63f19d0a651 by Victor Stinner in branch 'default':
Issue #18664, #18408: Rewrite PyErr_WriteUnraisable() to handle errors
http://hg.python.org/cpython/rev/e63f19d0a651

New changeset 8fb3a6f9b0a4 by Victor Stinner in branch 'default':
Restore changeset 5bd9db528aed (issue #18408)
http://hg.python.org/cpython/rev/8fb3a6f9b0a4
msg201593 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-29 00:19
New changeset 4ef4578db38a by Victor Stinner in branch 'default':
Issue #18408: Add a new PyFrame_FastToLocalsWithError() function to handle
http://hg.python.org/cpython/rev/4ef4578db38a
msg201594 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2013-10-29 00:20
Ok, most critical bugs have been fixed. If I find new bugs, I will open new issues.
msg201596 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-29 00:46
New changeset e1d51c42e5a1 by Victor Stinner in branch 'default':
Issue #18408: Fix PyUnicode_AsUTF8AndSize(), raise MemoryError exception on
http://hg.python.org/cpython/rev/e1d51c42e5a1

New changeset 7f4a976829f1 by Victor Stinner in branch 'default':
Issue #18408: Fix zipimport, handle PyUnicode_Substring() and get_subname() failures
http://hg.python.org/cpython/rev/7f4a976829f1
msg201599 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-29 01:44
New changeset 4749c3ac0654 by Victor Stinner in branch 'default':
Issue #18408: Fix iobase_readline(), handle PyByteArray_Resize() failure
http://hg.python.org/cpython/rev/4749c3ac0654
msg201601 - (view) Author: Roundup Robot (python-dev) Date: 2013-10-29 02:51
New changeset 549cd815bf8d by Victor Stinner in branch 'default':
Issue #18408: Fix error handling in PyBytes_FromObject()
http://hg.python.org/cpython/rev/549cd815bf8d

New changeset 9d9371d4ff7b by Victor Stinner in branch 'default':
Issue #18408: Fix bytearrayiter.partition()/rpartition(), handle
http://hg.python.org/cpython/rev/9d9371d4ff7b

New changeset 20cd15a28ad3 by Victor Stinner in branch 'default':
Issue #18408: Fix _ctypes_alloc_format_string(), raise MemoryError on memory
http://hg.python.org/cpython/rev/20cd15a28ad3

New changeset 1386fb31c0ed by Victor Stinner in branch 'default':
Issue #18408: Fix PyCStructUnionType_update_stgdict(), handle
http://hg.python.org/cpython/rev/1386fb31c0ed
msg204000 - (view) Author: Roundup Robot (python-dev) Date: 2013-11-23 13:13
New changeset 8f556ee0f6ba by Antoine Pitrou in branch 'default':
Fix refleak introduced by 4f730c045f5f (issue #18408) and unveiled by 95eea8624d05 (issue #16596).
http://hg.python.org/cpython/rev/8f556ee0f6ba
History
Date User Action Args
2013-11-23 13:13:08python-devsetmessages: + msg204000
2013-10-29 02:51:42python-devsetmessages: + msg201601
2013-10-29 01:44:03python-devsetmessages: + msg201599
2013-10-29 00:46:41python-devsetmessages: + msg201596
2013-10-29 00:20:51hayposetstatus: open -> closed
resolution: fixed
messages: + msg201594
2013-10-29 00:19:57python-devsetmessages: + msg201593
2013-08-26 12:06:56python-devsetmessages: + msg196199
2013-08-23 21:55:13pitrousetmessages: + msg196042
2013-08-23 21:54:02hayposetmessages: + msg196041
2013-08-23 21:19:33pitrousetnosy: + pitrou
messages: + msg196030
2013-08-23 21:18:47python-devsetmessages: + msg196029
2013-08-14 00:19:49hayposetmessages: + msg195113
2013-08-14 00:11:36hayposetmessages: + msg195112
2013-08-01 22:49:43python-devsetmessages: + msg194129
2013-07-21 11:15:04hayposetmessages: + msg193438
2013-07-21 11:09:24python-devsetmessages: + msg193435
2013-07-19 22:01:27hayposetmessages: + msg193382
2013-07-19 21:41:10scodersetnosy: - scoder
2013-07-19 21:39:39hayposetmessages: + msg193378
2013-07-19 21:14:35python-devsetmessages: + msg193374
2013-07-19 09:11:26scodersetmessages: + msg193350
2013-07-18 23:43:33hayposetmessages: + msg193337
2013-07-18 21:43:44python-devsetmessages: + msg193329
2013-07-18 21:07:03hayposetmessages: + msg193327
2013-07-18 20:46:48python-devsetmessages: + msg193324
2013-07-18 19:49:24scodersetnosy: + scoder
messages: + msg193316
2013-07-18 01:08:26hayposetmessages: + msg193265
2013-07-18 01:08:06hayposetmessages: + msg193264
2013-07-18 00:51:10hayposetmessages: + msg193263
2013-07-18 00:31:43python-devsetmessages: + msg193262
2013-07-17 23:57:33python-devsetmessages: + msg193261
2013-07-17 23:46:16python-devsetmessages: + msg193258
2013-07-17 20:33:57python-devsetmessages: + msg193250
2013-07-17 20:11:48python-devsetmessages: + msg193245
2013-07-17 11:45:54hayposetmessages: + msg193225
2013-07-17 11:43:33python-devsetmessages: + msg193224
2013-07-17 10:24:11hayposetmessages: + msg193219
2013-07-17 10:12:14hayposetmessages: + msg193218
2013-07-17 08:51:48serhiy.storchakasetmessages: + msg193213
2013-07-17 08:47:46neologixsetnosy: + neologix
messages: + msg193211
2013-07-17 07:33:04serhiy.storchakasetmessages: + msg193209
2013-07-16 23:24:05hayposetmessages: + msg193202
2013-07-16 23:12:10hayposetmessages: + msg193200
2013-07-16 23:10:20python-devsetmessages: + msg193199
2013-07-16 23:01:22python-devsetmessages: + msg193198
2013-07-16 22:28:48serhiy.storchakasetassignee: haypo

messages: + msg193196
nosy: + serhiy.storchaka
2013-07-16 21:09:14python-devsetmessages: + msg193192
2013-07-16 01:32:37hayposetmessages: + msg193144
2013-07-16 01:28:49hayposetmessages: + msg193142
2013-07-16 00:33:08hayposetmessages: + msg193140
2013-07-16 00:02:12python-devsetmessages: + msg193138
2013-07-15 22:12:45python-devsetmessages: + msg193132
2013-07-15 22:08:16hayposetfiles: + frame_fasttolocals.patch

messages: + msg193131
2013-07-15 22:06:07hayposetfiles: + listpop.patch

messages: + msg193130
2013-07-12 00:07:50python-devsetmessages: + msg192920
2013-07-11 23:42:52python-devsetmessages: + msg192918
2013-07-11 22:59:04python-devsetmessages: + msg192917
2013-07-11 21:47:01python-devsetmessages: + msg192913
2013-07-11 21:12:08python-devsetmessages: + msg192908
2013-07-11 20:46:46python-devsetmessages: + msg192902
2013-07-10 12:04:12python-devsetmessages: + msg192797
2013-07-09 21:02:22gregory.p.smithsetnosy: + gregory.p.smith
2013-07-09 00:03:50hayposetfiles: + misc_fixes.patch

messages: + msg192706
2013-07-09 00:01:12hayposetfiles: + type_slot_tp_str.patch

messages: + msg192705
2013-07-09 00:00:28hayposetfiles: + call_exc_trace.patch

messages: + msg192704
2013-07-08 23:59:54hayposetfiles: + use_failmalloc.patch
keywords: + patch
messages: + msg192703
2013-07-08 22:53:46python-devsetmessages: + msg192701
2013-07-08 20:50:08hayposetmessages: + msg192694
2013-07-08 20:36:31python-devsetnosy: + python-dev
messages: + msg192693
2013-07-08 20:10:43haypocreate