Issue24913
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2015-08-21 23:09 by JohnLeitch, last changed 2022-04-11 14:58 by admin. This issue is now closed.
Files | ||||
---|---|---|---|---|
File name | Uploaded | Description | Edit | |
newblock_Uninitialized_variable.patch | JohnLeitch, 2015-08-21 23:08 | patch | ||
newblock_Uninitialized_variable.py | JohnLeitch, 2015-08-21 23:10 | repro | ||
fix_deque_overrun.diff | rhettinger, 2015-08-26 06:16 | Patch to fix overrun | review | |
fix_deque_overrun2.diff | rhettinger, 2015-08-26 08:52 | Revised patch to fix overrun | review |
Messages (15) | |||
---|---|---|---|
msg248985 - (view) | Author: John Leitch (JohnLeitch) * | Date: 2015-08-21 23:08 | |
Python 3.5 suffers from a vulnerability caused by the behavior of the newblock() function used by the collections.deque module. When called, newblock() allocates memory using PyMem_Malloc() and does not initialize it: static block * newblock(Py_ssize_t len) { block *b; if (len >= MAX_DEQUE_LEN) { PyErr_SetString(PyExc_OverflowError, "cannot add more blocks to the deque"); return NULL; } if (numfreeblocks) { numfreeblocks--; return freeblocks[numfreeblocks]; } b = PyMem_Malloc(sizeof(block)); <<<< Memory allocation. if (b != NULL) { return b; <<<< Buffer returned without initialization. } PyErr_NoMemory(); return NULL; } Because PyMem_Malloc does not initialize the memory, the block may contain garbage data. In some cases, this can lead to memory corruption which could be exploitable to achieve code execution. The following exception analysis is an example of EIP corruption: ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* *** The OS name list needs to be updated! Unknown Windows version: 10.0 *** FAULTING_IP: python35!PyUnicode_Type+0 696f60d8 a800 test al,0 EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 696f60d8 (python35!PyUnicode_Type) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000008 Parameter[1]: 696f60d8 Attempt to execute non-executable address 696f60d8 CONTEXT: 00000000 -- (.cxr 0x0;r) eax=696f60d8 ebx=00000002 ecx=00d9492c edx=00000002 esi=019b4e58 edi=0337b970 eip=696f60d8 esp=00bcf7dc ebp=00bcf7fc iopl=0 nv up ei pl nz na pe nc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 python35!PyUnicode_Type: 696f60d8 a800 test al,0 PROCESS_NAME: pythonw.exe ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_PARAMETER1: 00000008 EXCEPTION_PARAMETER2: 696f60d8 WRITE_ADDRESS: 696f60d8 FOLLOWUP_IP: python35!PyUnicode_Type+0 696f60d8 a800 test al,0 FAILED_INSTRUCTION_ADDRESS: python35!PyUnicode_Type+0 696f60d8 a800 test al,0 APP: pythonw.exe ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre FAULTING_THREAD: 000009dc DEFAULT_BUCKET_ID: SOFTWARE_NX_FAULT_CODE PRIMARY_PROBLEM_CLASS: SOFTWARE_NX_FAULT_CODE BUGCHECK_STR: APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_SOFTWARE_NX_FAULT_FALSE_POSITIVE LAST_CONTROL_TRANSFER: from 69505ad3 to 696f60d8 STACK_TEXT: 00bcf7fc 69505ad3 00000002 00bcf840 694253fc python35!PyUnicode_Type 00bcf808 694253fc 0337b970 019b4e58 00000002 python35!PyObject_RichCompare+0x53 00bcf840 695031c3 03a1a8f0 03a21878 00f83340 python35!deque_index+0xac 00bcf85c 69564433 03a21120 03a21878 00000000 python35!PyCFunction_Call+0x113 00bcf890 695618d8 00e23a08 00000000 00000040 python35!call_function+0x303 00bcf908 6956339f 00e23a08 00000000 00f83000 python35!PyEval_EvalFrameEx+0x2318 00bcf954 6959a142 00e40f58 00000000 00000000 python35!_PyEval_EvalCodeWithName+0x82f 00bcf990 69599fd5 00e40f58 00e40f58 00bcfa5c python35!run_mod+0x42 00bcf9bc 6959904a 00f801f0 00e366f0 00000101 python35!PyRun_FileExFlags+0x85 00bcfa00 6946f037 00f801f0 00e366f0 00000001 python35!PyRun_SimpleFileExFlags+0x20a 00bcfa2c 6946f973 00bcfa5c 00000000 6ecb2100 python35!run_file+0xe7 00bcfad4 1ce31279 00000002 00f79eb0 1ce3c588 python35!Py_Main+0x913 00bcfae4 1ce3145f 1ce30000 00000000 00f71c68 pythonw!wWinMain+0x19 00bcfb30 74ed3744 7f174000 74ed3720 5c8b59d2 pythonw!__scrt_common_main_seh+0xfd 00bcfb44 775aa064 7f174000 a81800d2 00000000 kernel32!BaseThreadInitThunk+0x24 00bcfb8c 775aa02f ffffffff 775cd7c3 00000000 ntdll!__RtlUserThreadStart+0x2f 00bcfb9c 00000000 1ce3150a 7f174000 00000000 ntdll!_RtlUserThreadStart+0x1b STACK_COMMAND: ~0s; .ecxr ; kb SYMBOL_STACK_INDEX: 0 SYMBOL_NAME: python35!PyUnicode_Type+0 FOLLOWUP_NAME: MachineOwner MODULE_NAME: python35 IMAGE_NAME: python35.dll DEBUG_FLR_IMAGE_TIMESTAMP: 5598ccc2 FAILURE_BUCKET_ID: SOFTWARE_NX_FAULT_CODE_c0000005_python35.dll!PyUnicode_Type BUCKET_ID: APPLICATION_FAULT_SOFTWARE_NX_FAULT_CODE_SOFTWARE_NX_FAULT_FALSE_POSITIVE_BAD_IP_python35!PyUnicode_Type+0 ANALYSIS_SOURCE: UM FAILURE_ID_HASH_STRING: um:software_nx_fault_code_c0000005_python35.dll!pyunicode_type FAILURE_ID_HASH: {aa94d074-8f9b-b618-df4f-eaad15f84370} Followup: MachineOwner --------- To fix the issue, it is recommended that newblock use PyMem_Calloc instead of PyMem_Malloc. A proposed patch has been attached. Credit: John Leitch (johnleitch@outlook.com), Bryce Darling (darlingbryce@gmail.com) |
|||
msg249126 - (view) | Author: Raymond Hettinger (rhettinger) * ![]() |
Date: 2015-08-25 15:12 | |
I'm find the "exception analysis" to be unreadable. Have you found any place in the deque code where the uninitialized memory actually gets accessed? |
|||
msg249130 - (view) | Author: John Leitch (JohnLeitch) * | Date: 2015-08-25 15:54 | |
The "exception analysis" is output from the WinDbg !analyze command run on a crash where access to the uninitialized memory ultimately corrupted the instruction pointer, leading to a data execution prevention crash. That's why the disassembly is junk--the IP is not pointing to valid instructions. This crash was provided as an example because it demonstrates that the issue is likely exploitable, and can probably be used to achieve code execution. Here is an example of a crash where execution halts immediately upon attempted to dereference a corrupted pointer. Note that the pointer is 0xC0C0C0C0--a fill pattern indicative of uninitialized memory. 0:000> r eax=000002a2 ebx=551160a8 ecx=c0c0c0c0 edx=07e538e0 esi=07e538e0 edi=c0c0c0c0 eip=54f25a55 esp=004cf6e4 ebp=004cf6f4 iopl=0 nv up ei ng nz na pe cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287 python35!do_richcompare+0x15: 54f25a55 8b4704 mov eax,dword ptr [edi+4] ds:002b:c0c0c0c4=???????? 0:000> k ChildEBP RetAddr 004cf6f4 54f25be3 python35!do_richcompare+0x15 [c:\build\cpython\objects\object.c @ 659] 004cf700 54e453fc python35!PyObject_RichCompare+0x53 [c:\build\cpython\objects\object.c @ 718] (Inline) -------- python35!PyObject_RichCompareBool+0x14 [c:\build\cpython\objects\object.c @ 739] 004cf738 54f232d3 python35!deque_index+0xac [c:\build\cpython\modules\_collectionsmodule.c @ 933] 004cf754 54f8442f python35!PyCFunction_Call+0x113 [c:\build\cpython\objects\methodobject.c @ 109] 004cf788 54f818ec python35!call_function+0x2ff [c:\build\cpython\python\ceval.c @ 4651] 004cf800 54f8339f python35!PyEval_EvalFrameEx+0x232c [c:\build\cpython\python\ceval.c @ 3184] 004cf84c 54fba0b2 python35!_PyEval_EvalCodeWithName+0x82f [c:\build\cpython\python\ceval.c @ 3962] (Inline) -------- python35!PyEval_EvalCodeEx+0x21 [c:\build\cpython\python\ceval.c @ 3983] (Inline) -------- python35!PyEval_EvalCode+0x21 [c:\build\cpython\python\ceval.c @ 777] 004cf888 54fb9f45 python35!run_mod+0x42 [c:\build\cpython\python\pythonrun.c @ 970] 004cf8b4 54fb8fba python35!PyRun_FileExFlags+0x85 [c:\build\cpython\python\pythonrun.c @ 923] 004cf8f8 54e8f1f7 python35!PyRun_SimpleFileExFlags+0x20a [c:\build\cpython\python\pythonrun.c @ 396] (Inline) -------- python35!PyRun_AnyFileExFlags+0x4e [c:\build\cpython\python\pythonrun.c @ 80] 004cf924 54e8fb33 python35!run_file+0xe7 [c:\build\cpython\modules\main.c @ 318] 004cf9c8 1cd4143f python35!Py_Main+0x913 [c:\build\cpython\modules\main.c @ 768] (Inline) -------- python!invoke_main+0x1d [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 89] 004cfa14 75463744 python!__scrt_common_main_seh+0xff [f:\dd\vctools\crt\vcstartup\src\startup\exe_common.inl @ 264] 004cfa28 76f0a064 KERNEL32!BaseThreadInitThunk+0x24 004cfa70 76f0a02f ntdll!__RtlUserThreadStart+0x2f 004cfa80 00000000 ntdll!_RtlUserThreadStart+0x1b 0:000> !analyze -v -nodb ******************************************************************************* * * * Exception Analysis * * * ******************************************************************************* FAULTING_IP: python35!do_richcompare+15 [c:\build\cpython\objects\object.c @ 659] 54f25a55 8b4704 mov eax,dword ptr [edi+4] EXCEPTION_RECORD: ffffffff -- (.exr 0xffffffffffffffff) ExceptionAddress: 54f25a55 (python35!do_richcompare+0x00000015) ExceptionCode: c0000005 (Access violation) ExceptionFlags: 00000000 NumberParameters: 2 Parameter[0]: 00000000 Parameter[1]: c0c0c0c4 Attempt to read from address c0c0c0c4 CONTEXT: 00000000 -- (.cxr 0x0;r) eax=000002a2 ebx=551160a8 ecx=c0c0c0c0 edx=07e538e0 esi=07e538e0 edi=c0c0c0c0 eip=54f25a55 esp=004cf6e4 ebp=004cf6f4 iopl=0 nv up ei ng nz na pe cy cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010287 python35!do_richcompare+0x15: 54f25a55 8b4704 mov eax,dword ptr [edi+4] ds:002b:c0c0c0c4=???????? FAULTING_THREAD: 00004a48 DEFAULT_BUCKET_ID: INVALID_POINTER_READ PROCESS_NAME: python.exe ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s. EXCEPTION_PARAMETER1: 00000000 EXCEPTION_PARAMETER2: c0c0c0c4 READ_ADDRESS: c0c0c0c4 FOLLOWUP_IP: python35!do_richcompare+15 [c:\build\cpython\objects\object.c @ 659] 54f25a55 8b4704 mov eax,dword ptr [edi+4] NTGLOBALFLAG: 2000000 APPLICATION_VERIFIER_FLAGS: 0 APP: python.exe ANALYSIS_VERSION: 6.3.9600.17029 (debuggers(dbg).140219-1702) x86fre PRIMARY_PROBLEM_CLASS: INVALID_POINTER_READ BUGCHECK_STR: APPLICATION_FAULT_INVALID_POINTER_READ LAST_CONTROL_TRANSFER: from 54f25be3 to 54f25a55 STACK_TEXT: 004cf6f4 54f25be3 00000002 004cf738 54e453fc python35!do_richcompare+0x15 004cf700 54e453fc c0c0c0c0 07e538e0 00000002 python35!PyObject_RichCompare+0x53 004cf738 54f232d3 07e31d18 07e50e40 08e78b48 python35!deque_index+0xac 004cf754 54f8442f 08e78b48 07e50e40 00000000 python35!PyCFunction_Call+0x113 004cf788 54f818ec 0586eab0 00000000 00000040 python35!call_function+0x2ff 004cf800 54f8339f 0586eab0 00000000 08910ff0 python35!PyEval_EvalFrameEx+0x232c 004cf84c 54fba0b2 0588ff80 00000000 00000000 python35!_PyEval_EvalCodeWithName+0x82f 004cf888 54fb9f45 0588ff80 0588ff80 004cf954 python35!run_mod+0x42 004cf8b4 54fb8fba 06a90fc8 0581bc70 00000101 python35!PyRun_FileExFlags+0x85 004cf8f8 54e8f1f7 06a90fc8 0581bc70 00000001 python35!PyRun_SimpleFileExFlags+0x20a 004cf924 54e8fb33 004cf954 71902100 71902108 python35!run_file+0xe7 004cf9c8 1cd4143f 00000002 05b46f08 05b4cf48 python35!Py_Main+0x913 004cfa14 75463744 7ecee000 75463720 fbb4cf67 python!__scrt_common_main_seh+0xff 004cfa28 76f0a064 7ecee000 949593e0 00000000 KERNEL32!BaseThreadInitThunk+0x24 004cfa70 76f0a02f ffffffff 76f2d7ec 00000000 ntdll!__RtlUserThreadStart+0x2f 004cfa80 00000000 1cd414f7 7ecee000 00000000 ntdll!_RtlUserThreadStart+0x1b STACK_COMMAND: .cxr 0x0 ; kb FAULTING_SOURCE_LINE: c:\build\cpython\objects\object.c FAULTING_SOURCE_FILE: c:\build\cpython\objects\object.c FAULTING_SOURCE_LINE_NUMBER: 659 FAULTING_SOURCE_CODE: 655: PyObject *res; 656: int checked_reverse_op = 0; 657: 658: if (v->ob_type != w->ob_type && > 659: PyType_IsSubtype(w->ob_type, v->ob_type) && 660: (f = w->ob_type->tp_richcompare) != NULL) { 661: checked_reverse_op = 1; 662: res = (*f)(w, v, _Py_SwappedOp[op]); 663: if (res != Py_NotImplemented) 664: return res; SYMBOL_STACK_INDEX: 0 SYMBOL_NAME: python35!do_richcompare+15 FOLLOWUP_NAME: MachineOwner MODULE_NAME: python35 IMAGE_NAME: python35.dll DEBUG_FLR_IMAGE_TIMESTAMP: 55c83105 FAILURE_BUCKET_ID: INVALID_POINTER_READ_c0000005_python35.dll!do_richcompare BUCKET_ID: APPLICATION_FAULT_INVALID_POINTER_READ_python35!do_richcompare+15 ANALYSIS_SOURCE: UM FAILURE_ID_HASH_STRING: um:invalid_pointer_read_c0000005_python35.dll!do_richcompare FAILURE_ID_HASH: {9d923c37-6c51-89af-91c6-b0039172374e} Followup: MachineOwner --------- |
|||
msg249132 - (view) | Author: Stefan Krah (skrah) * ![]() |
Date: 2015-08-25 16:15 | |
I guess that in the test case the stop parameter is set to 4 in deque_index(), but it should be clamped to 3. |
|||
msg249181 - (view) | Author: Raymond Hettinger (rhettinger) * ![]() |
Date: 2015-08-26 06:19 | |
Larry, this may need to go into 3.5 if there is still an opportunity. |
|||
msg249197 - (view) | Author: Larry Hastings (larry) * ![]() |
Date: 2015-08-26 14:23 | |
Please create a pull request at your earliest convenience. |
|||
msg249201 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2015-08-26 15:10 | |
New changeset ae8ec66adc7f by Raymond Hettinger in branch '3.5': Issue #24913: Fix overrun error in deque.index(). https://hg.python.org/cpython/rev/ae8ec66adc7f |
|||
msg249202 - (view) | Author: Raymond Hettinger (rhettinger) * ![]() |
Date: 2015-08-26 15:12 | |
> Please create a pull request at your earliest convenience. I'm not sure what that entails. Can you just apply ae8ec66adc7f and take it from here? |
|||
msg249203 - (view) | Author: Larry Hastings (larry) * ![]() |
Date: 2015-08-26 15:19 | |
https://docs.python.org/devguide/devcycle.html#release-candidate-rc https://mail.python.org/pipermail/python-dev/2015-August/141167.html |
|||
msg249570 - (view) | Author: Larry Hastings (larry) * ![]() |
Date: 2015-09-02 20:50 | |
Assigning to Brett, who has agreed to do the merge to 3.5.0 that Raymond has declined to do. |
|||
msg249594 - (view) | Author: Raymond Hettinger (rhettinger) * ![]() |
Date: 2015-09-03 02:39 | |
Thanks Brett :-) |
|||
msg249678 - (view) | Author: Brett Cannon (brett.cannon) * ![]() |
Date: 2015-09-03 17:27 | |
PR created, Larry. |
|||
msg249684 - (view) | Author: Larry Hastings (larry) * ![]() |
Date: 2015-09-03 20:00 | |
Merged. Please do a (null) merge forward into 3.5.1 and 3.6. Thanks! |
|||
msg249700 - (view) | Author: Roundup Robot (python-dev) ![]() |
Date: 2015-09-03 22:36 | |
New changeset 9f8c59e61594 by Brett Cannon in branch '3.5': Issue #24913: Fix overrun error in deque.index(). https://hg.python.org/cpython/rev/9f8c59e61594 New changeset d093d87e449c by Brett Cannon in branch '3.5': Merge from 3.5.0 for issue #24913 https://hg.python.org/cpython/rev/d093d87e449c New changeset c6e0c29913ec by Brett Cannon in branch 'default': Merge from 3.5 for issue #24913 https://hg.python.org/cpython/rev/c6e0c29913ec |
|||
msg249701 - (view) | Author: Brett Cannon (brett.cannon) * ![]() |
Date: 2015-09-03 22:37 | |
OK, that should cover 3.5.0 and then null merge through 3.5 and default. I thus consider my favour to Larry done and Raymond now owes me one. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-11 14:58:20 | admin | set | nosy:
+ ned.deily github: 69101 |
2015-09-08 18:36:57 | Arfrever | set | nosy:
+ Arfrever |
2015-09-03 22:37:15 | brett.cannon | set | messages: + msg249701 |
2015-09-03 22:36:01 | python-dev | set | messages: + msg249700 |
2015-09-03 20:00:57 | larry | set | status: open -> closed resolution: fixed messages: + msg249684 |
2015-09-03 17:27:08 | brett.cannon | set | assignee: brett.cannon -> larry messages: + msg249678 stage: resolved |
2015-09-03 02:39:56 | rhettinger | set | messages: + msg249594 |
2015-09-02 20:50:40 | larry | set | assignee: larry -> brett.cannon messages: + msg249570 nosy: + brett.cannon |
2015-08-26 15:19:01 | larry | set | messages: + msg249203 |
2015-08-26 15:12:14 | rhettinger | set | messages: + msg249202 |
2015-08-26 15:10:18 | python-dev | set | nosy:
+ python-dev messages: + msg249201 |
2015-08-26 14:23:56 | larry | set | priority: high -> release blocker messages: + msg249197 |
2015-08-26 08:52:02 | rhettinger | set | files: + fix_deque_overrun2.diff |
2015-08-26 06:19:03 | rhettinger | set | priority: low -> high assignee: rhettinger -> larry title: newblock() Uninitialized Variable -> deque.index() overruns deque boundary nosy: + larry versions: + Python 3.6 messages: + msg249181 |
2015-08-26 06:16:14 | rhettinger | set | files: + fix_deque_overrun.diff |
2015-08-25 16:15:25 | skrah | set | nosy:
+ skrah messages: + msg249132 |
2015-08-25 15:54:54 | JohnLeitch | set | messages: + msg249130 |
2015-08-25 15:12:08 | rhettinger | set | priority: normal -> low messages: + msg249126 |
2015-08-25 15:05:21 | rhettinger | set | assignee: rhettinger |
2015-08-23 22:52:46 | brycedarling | set | nosy:
+ brycedarling |
2015-08-22 00:45:06 | r.david.murray | set | nosy:
+ rhettinger |
2015-08-21 23:10:29 | JohnLeitch | set | files: + newblock_Uninitialized_variable.py |
2015-08-21 23:09:06 | JohnLeitch | create |