This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: Py_BuildValue() can call Python code with an exception set
Type: Stage: commit review
Components: Versions: Python 3.4
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: python-dev, serhiy.storchaka, vstinner
Priority: normal Keywords: patch

Created on 2013-12-19 11:13 by vstinner, last changed 2022-04-11 14:57 by admin. This issue is now closed.

File name Uploaded Description Edit
py_buildvalue_failed.patch vstinner, 2013-12-19 11:13 review
parsermodule.patch vstinner, 2013-12-19 11:14 review
Messages (7)
msg206602 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-19 11:13
In Python 3.4, an assertion now checks that no exception is set when arbitrary Python code is called. Python code may suppress the exception.

When Py_BuildValue() is used to build a tuple and an error when the creation of an item failed, the function may execute indirectly Python code with an exception set.

Attached patch works around the issue by storing the current exception in a variable and then restore it. It changes the behaviour if more than one exception is raised: at the end, its the first exception which is passed to the caller. I prefer to get the first exception instead of the following exceptions, because following exceptions may be caused by the first exception. See for example the parser bug below for an example of a second exception caused by the first exception.


Example with the parser module:

            PyObject *err = Py_BuildValue("os", elem,
                                          "Illegal node construct.");
            PyErr_SetObject(parser_error, err);

The "o" is not a valid format and so a SystemError is raised, but then the UTF-8 decoder tries to raise a second exception, an UnicodeDecodeError, because the decoders is called with an invalid bytes string (elem variable, instead of the "Illegal node construct." string, because "o" format didn't increment the argument pointer). The bug is obvious in the parser module, but the problem is more general than that.

Gdb traceback of the parser bug:

python: Objects/typeobject.c:741: type_call: Assertion `!PyErr_Occurred()' failed.

Program received signal SIGABRT, Aborted.
0x000000301c0359e9 in raise () from /lib64/
Missing separate debuginfos, use: debuginfo-install glibc-2.17-19.fc19.x86_64
(gdb) where
#0  0x000000301c0359e9 in raise () from /lib64/
#1  0x000000301c0370f8 in abort () from /lib64/
#2  0x000000301c02e956 in __assert_fail_base () from /lib64/
#3  0x000000301c02ea02 in __assert_fail () from /lib64/
#4  0x00000000004dae18 in type_call (type=0x906780 <_PyExc_UnicodeDecodeError>, args=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'), kwds=0x0)
    at Objects/typeobject.c:741
#5  0x000000000045fb75 in PyObject_Call (func=<type at remote 0x906780>, arg=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'), kw=0x0)
    at Objects/abstract.c:2067
#6  0x000000000045fd0f in call_function_tail (callable=<type at remote 0x906780>, args=('utf-8', b'\xc8\x87\xe0\xf7\xff\x7f', 2, 3, 'invalid continuation byte'))
    at Objects/abstract.c:2104
#7  0x000000000045ff94 in _PyObject_CallFunction_SizeT (callable=<type at remote 0x906780>, format=0x669e19 "sy#nns") at Objects/abstract.c:2150
#8  0x000000000048643d in PyUnicodeDecodeError_Create (encoding=0x67e312 "utf-8", object=0x7ffff6ddcf40 "\310\207\340\367\377\177", length=6, start=2, end=3, 
    reason=0x67f1c1 "invalid continuation byte") at Objects/exceptions.c:1960
#9  0x000000000050fe7c in make_decode_exception (exceptionObject=0x7fffffff9090, encoding=0x67e312 "utf-8", input=0x7ffff6ddcf40 "\310\207\340\367\377\177", length=6, 
    startpos=2, endpos=3, reason=0x67f1c1 "invalid continuation byte") at Objects/unicodeobject.c:4009
#10 0x0000000000510015 in unicode_decode_call_errorhandler_writer (errors=0x0, errorHandler=0x7fffffff9098, encoding=0x67e312 "utf-8", 
    reason=0x67f1c1 "invalid continuation byte", input=0x7fffffff90b8, inend=0x7fffffff90b0, startinpos=0x7fffffff90a8, endinpos=0x7fffffff90a0, 
    exceptionObject=0x7fffffff9090, inptr=0x7fffffff9088, writer=0x7fffffff90c0) at Objects/unicodeobject.c:4157
#11 0x00000000005163a8 in PyUnicode_DecodeUTF8Stateful (s=0x7ffff6ddcf42 "\340\367\377\177", size=6, errors=0x0, consumed=0x0) at Objects/unicodeobject.c:4798
#12 0x0000000000506198 in PyUnicode_FromStringAndSize (u=0x7ffff6ddcf40 "\310\207\340\367\377\177", size=6) at Objects/unicodeobject.c:1840
#13 0x00000000005da9d2 in do_mkvalue (p_format=0x7fffffff92e0, p_va=0x7fffffff92c8, flags=0) at Python/modsupport.c:319
#14 0x00000000005d9e50 in do_mktuple (p_format=0x7fffffff92e0, p_va=0x7fffffff92c8, endchar=0, n=2, flags=0) at Python/modsupport.c:164
#15 0x00000000005db067 in va_build_value (format=0x7ffff750f34b "os", va=0x7fffffff9310, flags=0) at Python/modsupport.c:455
#16 0x00000000005dae8c in Py_BuildValue (format=0x7ffff750f34b "os") at Python/modsupport.c:410
#17 0x00007ffff750641c in build_node_children (tuple=(1467, -415088696226, 183475025091, 3, -85006798, 915338703290779849), root=0x7ffff7f7f6b8, line_num=0x7fffffff95cc)
    at /home/haypo/prog/python/default/Modules/parsermodule.c:788
msg206603 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-12-19 11:14
parsermodule.patch: fix usage of Py_BuildValue() in the parser module.
msg208682 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2014-01-21 18:21

I have searched the sources and have not found any use of Py_BuildValue() which can be used for test. Py_BuildValue() can fail only due to memory error. It can also can fail with non-UTF8 string, but all uses of Py_BuildValue() with a string in a tuple proceeds only ASCII strings.
msg208692 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-21 19:53
parsermodule.patch was applied as 91cb83f895cf (Python 3.3) and cd952e92c180 (3.4).
msg208693 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2014-01-21 19:55
New changeset 29b4eb47f65e by Victor Stinner in branch 'default':
Issue #20024: Py_BuildValue() now saves/restores the current exception before
msg208695 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-01-21 19:56
Thanks for the review Serhiy.
msg259253 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-01-30 11:25
Committed patch fixed the issue only for tuples, but not for lists or dicts. See more general patch in issue26168.
Date User Action Args
2022-04-11 14:57:55adminsetgithub: 64223
2016-01-30 11:25:58serhiy.storchakasetmessages: + msg259253
2014-01-21 19:56:19vstinnersetstatus: open -> closed
resolution: fixed
messages: + msg208695
2014-01-21 19:55:49python-devsetnosy: + python-dev
messages: + msg208693
2014-01-21 19:53:57vstinnersetmessages: + msg208692
2014-01-21 18:21:53serhiy.storchakasetmessages: + msg208682
stage: test needed -> commit review
2013-12-19 15:56:50serhiy.storchakasetstage: test needed
2013-12-19 11:14:39vstinnersetmessages: + msg206603
2013-12-19 11:14:21vstinnersetfiles: + parsermodule.patch
2013-12-19 11:13:30vstinnercreate