This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Author neologix
Recipients Thomas.Smith, liang, neologix
Date 2010-04-16.21:58:59
SpamBayes Score 1.1657342e-15
Marked as misclassified No
Message-id <1271455142.75.0.373735125193.issue7332@psf.upfronthosting.co.za>
In-reply-to
Content
It's definitely a stack overflow.
Most of the backtraces show an important number of frames.
The last frame is this:
#0  PyMarshal_ReadLastObjectFromFile (fp=0x13e8200)
    at ../Python/marshal.c:1026
	filesize = <value optimized out>

and a disassembly show us that the segfault is generated on a callq:
0x4bd4d6 <PyMarshal_ReadLastObjectFromFile+54>:	callq  0x4168e8 <fileno@plt>

And if you look at the code, it's obvious what's happening:
PyObject *
PyMarshal_ReadLastObjectFromFile(FILE *fp)
{
/* 75% of 2.1's .pyc files can exploit SMALL_FILE_LIMIT.
 * REASONABLE_FILE_LIMIT is by defn something big enough for Tkinter.pyc.
 */
#define SMALL_FILE_LIMIT (1L << 14)
#define REASONABLE_FILE_LIMIT (1L << 18)
#ifdef HAVE_FSTAT
	off_t filesize;
#endif
#ifdef HAVE_FSTAT
	filesize = getfilesize(fp);
	if (filesize > 0) {
		char buf[SMALL_FILE_LIMIT];
		char* pBuf = NULL;
		if (filesize <= SMALL_FILE_LIMIT)
			pBuf = buf;
		else if (filesize <= REASONABLE_FILE_LIMIT)
			pBuf = (char *)PyMem_MALLOC(filesize);
		if (pBuf != NULL) {
[...]
}

SMALL_FILE_LIMIT is 1 << 14 which is roughly 16K (not that reasonable :-).
So when we enter PyMarshal_ReadLastObjectFromFile and allocate buf, we push around 16K on the stack, which is a lot. That's why we segfault soon after when we call a function (callq), there's no space left on the stack.
So there are several solutions:
- make buf static, but it would increase Python size by almost 16K
- reduce SMALL_FILE_LIMIT, or remove it altogether. I guess SMALL_FILE_LIMIT is there to speedup loading of small files, but I'm not sure that malloc() would incur an important overhead
- reading the whole file to memory sounds weird, we sould probably be using mmap() here

Peers ?
History
Date User Action Args
2010-04-16 21:59:02neologixsetrecipients: + neologix, liang, Thomas.Smith
2010-04-16 21:59:02neologixsetmessageid: <1271455142.75.0.373735125193.issue7332@psf.upfronthosting.co.za>
2010-04-16 21:59:01neologixlinkissue7332 messages
2010-04-16 21:58:59neologixcreate