classification
Title: null pointer deref and segfault in _PyObject_Alloc (obmalloc.c:1258)
Type: crash Stage:
Components: Interpreter Core Versions: Python 3.7
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: benjamin.peterson, geeknik, haypo, inada.naoki
Priority: normal Keywords:

Created on 2017-08-10 02:04 by geeknik, last changed 2017-08-10 09:03 by haypo.

Files
File name Uploaded Description Edit
31166.txt geeknik, 2017-08-10 07:51 valgrind -q ./python 31166.py
Messages (9)
msg300034 - (view) Author: geeknik (geeknik) Date: 2017-08-10 02:04
Python 3.7 git commit 3ca9f50 compiled with afl-clang-fast on Ubuntu 16 x64. The following script triggers undefined-behavior followed by a null pointer dereference and a segfault.


import gc
t0ing0=object()
class A(object):
    def f():0
    x=t0ing0
r=gc.get_referrers(t0ing0)
if[0]:dct=r[0]
a=A
for i in range(1):a.f
dct["f"]=lambda:0
(a.f)


Objects/dictobject.c:547:12: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:547:12 in
Objects/dictobject.c:1105:18: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1105:18 in
Objects/dictobject.c:2739:15: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:2739:15 in
Objects/dictobject.c:789:27: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:789:27 in
Objects/dictobject.c:1104:18: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1104:18 in
Objects/dictobject.c:994:15: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:994:15 in
Objects/dictobject.c:683:11: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:683:11 in
Objects/dictobject.c:1024:9: runtime error: index 64 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1024:9 in
Objects/dictobject.c:2882:31: runtime error: index 64 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:2882:31 in
Objects/dictobject.c:2346:15: runtime error: index 128 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:2346:15 in
Objects/dictobject.c:1449:11: runtime error: index 32 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1449:11 in
Objects/dictobject.c:744:27: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:744:27 in
Objects/dictobject.c:1631:22: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1631:22 in
Objects/dictobject.c:554:31: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:554:31 in
Objects/dictobject.c:1183:15: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:1183:15 in
Objects/dictobject.c:835:27: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:835:27 in
Objects/dictobject.c:2036:10: runtime error: index 128 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:2036:10 in
Objects/dictobject.c:3504:38: runtime error: index 16 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:3504:38 in
Objects/dictobject.c:3422:38: runtime error: index 64 out of bounds for type 'int8_t [8]'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/dictobject.c:3422:38 in
Objects/obmalloc.c:1258:36: runtime error: load of misaligned address 0xffffffffffffffff for type 'block *' (aka 'unsigned char *'), which requires 8 byte alignment
0xffffffffffffffff: note: pointer points here
<memory cannot be printed>
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Objects/obmalloc.c:1258:36 in
ASAN:DEADLYSIGNAL
=================================================================
==768==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x000000506f9f bp 0x7fff0666cbc0 sp 0x7fff0666cb20 T0)
==768==The signal is caused by a READ memory access.
==768==Hint: address points to the zero page.
    #0 0x506f9e in _PyObject_Alloc /root/cpython/Objects/obmalloc.c:1258:36
    #1 0x9a669b in PyUnicode_New /root/cpython/Objects/unicodeobject.c:1296:24
    #2 0x9fea51 in _PyUnicodeWriter_PrepareInternal /root/cpython/Objects/unicodeobject.c:13561:26
    #3 0x9b9db4 in PyUnicode_DecodeUTF8Stateful /root/cpython/Objects/unicodeobject.c:4995:9
    #4 0x9c0e65 in _PyUnicode_FromId /root/cpython/Objects/unicodeobject.c:2115:22
    #5 0x89561e in _PyObject_GetAttrId /root/cpython/Objects/object.c:850:23
    #6 0x6b3f4a in _PyObject_CallMethodId /root/cpython/Objects/call.c:1086:16
    #7 0x518569 in flush_std_files /root/cpython/Python/pylifecycle.c:889:15
    #8 0x517942 in Py_FinalizeEx /root/cpython/Python/pylifecycle.c:959:9
    #9 0x5a882b in Py_Main /root/cpython/Modules/main.c:921:9
    #10 0x500382 in main /root/cpython/./Programs/python.c:102:11
    #11 0x7f94db0bb3f0 in __libc_start_main /build/glibc-mXZSwJ/glibc-2.24/csu/../csu/libc-start.c:291
    #12 0x433e49 in _start (/root/cpython/python+0x433e49)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /root/cpython/Objects/obmalloc.c:1258:36 in _PyObject_Alloc
==768==ABORTING
msg300050 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2017-08-10 06:49
Yeah, bad things happen when the underlying class dict, which isn't normally exposed to Python, is mutated.
msg300051 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2017-08-10 07:02
I think it's false positive of ASAN.

We have dynamically sized block.
https://github.com/python/cpython/blob/3b0f620c1a2a21272a9e2aeca6ca1d1ac10f8162/Objects/dict-common.h#L49-L69

dictobject.c:547 calls memcpy to fill the block and head pointer
is defined as `int8_t [8]`.
ASAN doesn't know this is overallocated memory block.
msg300058 - (view) Author: geeknik (geeknik) Date: 2017-08-10 07:51
So if I leave UBSan and ASan out of the equation and compile with gcc and run this script:

Program received signal SIGSEGV, Segmentation fault.
update_refs (containers=<optimized out>) at Modules/gcmodule.c:353
353             _PyGCHead_SET_REFS(gc, Py_REFCNT(FROM_GC(gc)));
(gdb) bt
#0  update_refs (containers=<optimized out>) at Modules/gcmodule.c:353
#1  collect (generation=generation@entry=2,
    n_collected=n_collected@entry=0x7fffffffe2f8,
    n_uncollectable=n_uncollectable@entry=0x7fffffffe300,
    nofail=nofail@entry=0) at Modules/gcmodule.c:962
#2  0x00005555555d5365 in collect_with_callback (generation=2)
    at Modules/gcmodule.c:1135
#3  PyGC_Collect () at Modules/gcmodule.c:1622
#4  _PyGC_CollectIfEnabled () at Modules/gcmodule.c:1635
#5  0x00005555555b8e28 in Py_FinalizeEx () at Python/pylifecycle.c:978
#6  0x00005555555b9225 in Py_FinalizeEx () at Python/pylifecycle.c:1119
#7  0x00005555555d2ed2 in Py_Main (argc=<optimized out>, argv=<optimized out>)
    at Modules/main.c:921
#8  0x00005555555aa1cb in main (argc=2, argv=<optimized out>)
    at ./Programs/python.c:102
(gdb) list
348     update_refs(PyGC_Head *containers)
349     {
350         PyGC_Head *gc = containers->gc.gc_next;
351         for (; gc != containers; gc = gc->gc.gc_next) {
352             assert(_PyGCHead_REFS(gc) == GC_REACHABLE);
353             _PyGCHead_SET_REFS(gc, Py_REFCNT(FROM_GC(gc)));
354             /* Python's cyclic gc should never see an incoming refcount
355              * of 0:  if something decref'ed to 0, it should have been
356              * deallocated immediately at that time.
357              * Possible cause (if the assert triggers):  a tp_dealloc

Valgrind shows a null deref as well after some invalid reads and conditional jumps. I've attached the log, it's a bit verbose.
msg300061 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2017-08-10 08:27
As Benjamin commented, this is caused by mutating internal dict.

PyType_Lookup() use "method cache", based on "tp_version_tag" in the type object.
When you modify internal dict directly, namespace is changed without
invalidating tp_version_tag.
So cached pointer is used, and it's already deallocated.

I don't know we should fix it or not.
I don't have any idea fix this without any performance penalty.
msg300062 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 08:38
> I don't know we should fix it or not. I don't have any idea fix this without any performance penalty.

The PEP 509 (dict version) might help if we want to fix this bug.
msg300063 - (view) Author: INADA Naoki (inada.naoki) * (Python committer) Date: 2017-08-10 08:49
But we should check dicts of all parents.
It will has significant penalty, especially for classes having long mro (inheriting metaclass from typing module cause long mro).
msg300064 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 09:02
"But we should check dicts of all parents. It will has significant penalty, especially for classes having long mro (inheriting metaclass from typing module cause long mro)."

Oh right. That would defeat the whole purpose of the cache.

Maybe we should not fix the bug. You are not supposed to access the hidden dictionary :-)
msg300065 - (view) Author: STINNER Victor (haypo) * (Python committer) Date: 2017-08-10 09:03
Another solution is to replace class dict with a special type which invalidates the type cache on dict[key]=value.
History
Date User Action Args
2017-08-10 09:03:27hayposetmessages: + msg300065
2017-08-10 09:02:41hayposetmessages: + msg300064
2017-08-10 08:49:51inada.naokisetmessages: + msg300063
2017-08-10 08:38:57hayposetnosy: + haypo
messages: + msg300062
2017-08-10 08:27:17inada.naokisetmessages: + msg300061
2017-08-10 07:51:30geekniksetfiles: + 31166.txt

messages: + msg300058
2017-08-10 07:02:32inada.naokisetnosy: + inada.naoki
messages: + msg300051
2017-08-10 06:49:53benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg300050
2017-08-10 02:04:37geeknikcreate