In Python 3.4, an assertion now checks that no exception is set when arbitrary Python code is called. Python code may suppress the exception.

When Py_BuildValue() is used to build a tuple and an error when the creation of an item failed, the function may execute indirectly Python code with an exception set.

Attached patch works around the issue by storing the current exception in a variable and then restore it. It changes the behaviour if more than one exception is raised: at the end, its the first exception which is passed to the caller. I prefer to get the first exception instead of the following exceptions, because following exceptions may be caused by the first exception. See for example the parser bug below for an example of a second exception caused by the first exception.

--

Example with the parser module:

            PyObject *err = Py_BuildValue("os", elem,
                                          "Illegal node construct.");
            PyErr_SetObject(parser_error, err);

The "o" is not a valid format and so a SystemError is raised, but then the UTF-8 decoder tries to raise a second exception, an UnicodeDecodeError, because the decoders is called with an invalid bytes string (elem variable, instead of the "Illegal node construct." string, because "o" format didn't increment the argument pointer). The bug is obvious in the parser module, but the problem is more general than that.

Gdb traceback of the parser bug:

python: Objects/typeobject.c:741: type_call: Assertion `!PyErr_Occurred()' failed.

Program received signal SIGABRT, Aborted.
0x000000301c0359e9 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install glibc-2.17-19.fc19.x86_64
(gdb) where
#0  0x000000301c0359e9 in raise () from /lib64/libc.so.6
#1  0x000000301c0370f8 in abort () from /lib64/libc.so.6
#2  0x000000301c02e956 in __assert_fail_base () from /lib64/libc.so.6
#3  0x000000301c02ea02 in __assert_fail () from /lib64/libc.so.6
#4  0x00000000004dae18 in type_call (type=0x906780 <_PyExc_UnicodeDecodeError>, args=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'), kwds=0x0)
    at Objects/typeobject.c:741
#5  0x000000000045fb75 in PyObject_Call (func=<type at remote 0x906780>, arg=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'), kw=0x0)
    at Objects/abstract.c:2067
#6  0x000000000045fd0f in call_function_tail (callable=<type at remote 0x906780>, args=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'))
    at Objects/abstract.c:2104
#7  0x000000000045ff94 in _PyObject_CallFunction_SizeT (callable=<type at remote 0x906780>, format=0x669e19 "sy#nns") at Objects/abstract.c:2150
#8  0x000000000048643d in PyUnicodeDecodeError_Create (encoding=0x67e312 "utf-8", object=0x7ffff6ddcf40 "\310\207\340\367\377\177", length=6, start=2, end=3, 
    reason=0x67f1c1 "invalid continuation byte") at Objects/exceptions.c:1960
#9  0x000000000050fe7c in make_decode_exception (exceptionObject=0x7fffffff9090, encoding=0x67e312 "utf-8", input=0x7ffff6ddcf40 "\310\207\340\367\377\177", length=6, 
    startpos=2, endpos=3, reason=0x67f1c1 "invalid continuation byte") at Objects/unicodeobject.c:4009
#10 0x0000000000510015 in unicode_decode_call_errorhandler_writer (errors=0x0, errorHandler=0x7fffffff9098, encoding=0x67e312 "utf-8", 
    reason=0x67f1c1 "invalid continuation byte", input=0x7fffffff90b8, inend=0x7fffffff90b0, startinpos=0x7fffffff90a8, endinpos=0x7fffffff90a0, 
    exceptionObject=0x7fffffff9090, inptr=0x7fffffff9088, writer=0x7fffffff90c0) at Objects/unicodeobject.c:4157
#11 0x00000000005163a8 in PyUnicode_DecodeUTF8Stateful (s=0x7ffff6ddcf42 "\340\367\377\177", size=6, errors=0x0, consumed=0x0) at Objects/unicodeobject.c:4798
#12 0x0000000000506198 in PyUnicode_FromStringAndSize (u=0x7ffff6ddcf40 "\310\207\340\367\377\177", size=6) at Objects/unicodeobject.c:1840
#13 0x00000000005da9d2 in do_mkvalue (p_format=0x7fffffff92e0, p_va=0x7fffffff92c8, flags=0) at Python/modsupport.c:319
#14 0x00000000005d9e50 in do_mktuple (p_format=0x7fffffff92e0, p_va=0x7fffffff92c8, endchar=0, n=2, flags=0) at Python/modsupport.c:164
#15 0x00000000005db067 in va_build_value (format=0x7ffff750f34b "os", va=0x7fffffff9310, flags=0) at Python/modsupport.c:455
#16 0x00000000005dae8c in Py_BuildValue (format=0x7ffff750f34b "os") at Python/modsupport.c:410
#17 0x00007ffff750641c in build_node_children (tuple=(1467, -415088696226, 183475025091, 3, -85006798, 915338703290779849), root=0x7ffff7f7f6b8, line_num=0x7fffffff95cc)
    at /home/haypo/prog/python/default/Modules/parsermodule.c:788

parsermodule.patch: fix usage of Py_BuildValue() in the parser module.

LGTM.

I have searched the sources and have not found any use of Py_BuildValue() which can be used for test. Py_BuildValue() can fail only due to memory error. It can also can fail with non-UTF8 string, but all uses of Py_BuildValue() with a string in a tuple proceeds only ASCII strings.

parsermodule.patch was applied as 91cb83f895cf (Python 3.3) and cd952e92c180 (3.4).

New changeset 29b4eb47f65e by Victor Stinner in branch 'default':
Issue #20024: Py_BuildValue() now saves/restores the current exception before
http://hg.python.org/cpython/rev/29b4eb47f65e

Thanks for the review Serhiy.

Committed patch fixed the issue only for tuples, but not for lists or dicts. See more general patch in issue26168.