classification
Title: Error on test_issue_1395_5 with Python 2.7 and VS2010
Type: crash Stage: resolved
Components: IO Versions: Python 3.3, Python 3.4, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: serhiy.storchaka Nosy List: Yogesh.Chaudhari, amaury.forgeotdarc, brian.curtin, loewis, pitrou, prlw1, python-dev, sable, serhiy.storchaka
Priority: high Keywords: easy, patch

Created on 2011-11-23 14:02 by sable, last changed 2013-08-20 18:24 by serhiy.storchaka. This issue is now closed.

Files
File name Uploaded Description Edit
issue13461-27.patch Yogesh.Chaudhari, 2013-05-12 20:29 review
issue13461-3x.patch Yogesh.Chaudhari, 2013-05-12 20:57 similar patch for 3.x to not use numerical literals review
Messages (10)
msg148184 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-23 14:02
I am trying to get Python working when compiled with Visual Studio 2010 (cf issue 13210).

When running the tests with the python 2.7 branch compiled with VS2010, the "test_issue_1395_5" in test_io.py will cause Python to eat the whole memory within a few seconds and make the server completely unresponsive.
msg148186 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-23 15:29
The problem is in CTextIOWrapperTest.test_issue1395_5
Here is the backtrace:

 	msvcr100d.dll!memset()  Line 145	Asm
>	msvcr100d.dll!_heap_alloc_dbg_impl(unsigned __int64 nSize, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp)  Line 498	C++
 	msvcr100d.dll!_nh_malloc_dbg_impl(unsigned __int64 nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp)  Line 239 + 0x22 bytes	C++
 	msvcr100d.dll!_nh_malloc_dbg(unsigned __int64 nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine)  Line 302 + 0x2a bytes	C++
 	msvcr100d.dll!malloc(unsigned __int64 nSize)  Line 56 + 0x21 bytes	C++
 	python27_d.dll!PyObject_Malloc(unsigned __int64 nbytes)  Line 944	C
 	python27_d.dll!_PyObject_DebugMallocApi(char id, unsigned __int64 nbytes)  Line 1445 + 0xa bytes	C
 	python27_d.dll!_PyObject_DebugMalloc(unsigned __int64 nbytes)  Line 1413	C
 	python27_d.dll!PyString_FromStringAndSize(const char * str, __int64 size)  Line 88 + 0x11 bytes	C
 	python27_d.dll!do_mkvalue(const char * * p_format, char * * p_va, int flags)  Line 427 + 0xf bytes	C
 	python27_d.dll!va_build_value(const char * format, char * va, int flags)  Line 537 + 0x14 bytes	C
 	python27_d.dll!_Py_VaBuildValue_SizeT(const char * format, char * va)  Line 511	C
 	python27_d.dll!_PyObject_CallMethod_SizeT(_object * o, char * name, char * format, ...)  Line 2671 + 0xf bytes	C
 	python27_d.dll!textiowrapper_tell(textio * self, _object * args)  Line 2222 + 0x2c bytes	C



So the problem happens when calling in textio.c:
{{{
PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, 1);
}}}

self->decoder is of type "_io.IncrementalNewlineDecoder" and input is "BBB".

This will result in PyString_FromStringAndSize being called with size = 4294967297, which will cause the server to fall.
msg148257 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2011-11-24 14:39
What if you replace:

PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, 1);

with:

PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, (Py_ssize_t) 1);
msg148327 - (view) Author: Sébastien Sablé (sable) Date: 2011-11-25 15:38
Thanks Antoine! It solved the issue.

I will check soon with Python trunk to see if the same thing applies.
msg148328 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2011-11-25 15:52
I've identified a few other cases where a '#' format is passed a numeric literal:

Python/codecs.c:514: return Py_BuildValue("(u#n)", &end, 0, end);
Modules/_io/textio.c:2323: DECODER_DECODE(input, 1, n);
msg182102 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-02-14 15:42
Can we fix this easy issue before 2.7.4 release?
msg182103 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-02-14 15:44
Ageed, it's probably easy enough.
msg189061 - (view) Author: Yogesh Chaudhari (Yogesh.Chaudhari) * Date: 2013-05-12 20:29
This patch should work for 2.7 branch
msg195704 - (view) Author: Roundup Robot (python-dev) Date: 2013-08-20 17:12
New changeset 826233404be8 by Serhiy Storchaka in branch '3.3':
Issue #13461: Fix a crash in the TextIOWrapper.tell method on 64-bit platforms.
http://hg.python.org/cpython/rev/826233404be8

New changeset 6c9d49b8e3ec by Serhiy Storchaka in branch 'default':
Issue #13461: Fix a crash in the TextIOWrapper.tell method on 64-bit platforms.
http://hg.python.org/cpython/rev/6c9d49b8e3ec

New changeset 5e679ef2a55c by Serhiy Storchaka in branch '2.7':
Issue #13461: Fix a crash in the TextIOWrapper.tell method and in the "replace"
http://hg.python.org/cpython/rev/5e679ef2a55c
msg195708 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2013-08-20 18:24
Thank you for the report Sébastien. Thank you for the patch Yogesh.
History
Date User Action Args
2013-08-20 18:24:04serhiy.storchakasetstatus: open -> closed
versions: - Python 3.2
messages: + msg195708

resolution: fixed
stage: needs patch -> resolved
2013-08-20 17:12:00python-devsetnosy: + python-dev
messages: + msg195704
2013-08-19 21:14:46serhiy.storchakasetassignee: serhiy.storchaka
2013-05-12 20:57:23Yogesh.Chaudharisetfiles: + issue13461-3x.patch
2013-05-12 20:41:22Yogesh.Chaudharisethgrepos: - hgrepo191
2013-05-12 20:29:34Yogesh.Chaudharisetfiles: + issue13461-27.patch

nosy: + Yogesh.Chaudhari
messages: + msg189061

hgrepos: + hgrepo191
keywords: + patch
2013-02-14 15:44:37pitrousetstage: needs patch
messages: + msg182103
versions: + Python 3.4
2013-02-14 15:42:21serhiy.storchakasetkeywords: + easy
nosy: + serhiy.storchaka
messages: + msg182102

2012-08-20 14:57:08prlw1setnosy: + prlw1
2011-11-25 15:52:25amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg148328
2011-11-25 15:48:11pitrousetpriority: normal -> high
type: crash
2011-11-25 15:48:05pitrousetversions: + Python 3.2, Python 3.3
2011-11-25 15:38:25sablesetmessages: + msg148327
2011-11-24 14:39:45pitrousetnosy: + loewis, brian.curtin, pitrou
messages: + msg148257
2011-11-23 15:29:47sablesetmessages: + msg148186
2011-11-23 14:02:38sablecreate