classification
Title: test_array fails on FreeBSD7 amd64
Type: crash Stage:
Components: Tests Versions: Python 3.0, Python 2.6, Python 2.5
process
Status: closed Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: aimacintyre, georg.brandl, loewis, mark.dickinson, robrien, terry.reedy
Priority: normal Keywords: patch

Created on 2008-09-14 01:30 by robrien, last changed 2013-03-28 10:00 by georg.brandl. This issue is now closed.

Files
File name Uploaded Description Edit
64bit_2.patch mark.dickinson, 2008-09-16 13:10 Possible fix? review
Messages (16)
msg73204 - (view) Author: Reed O'Brien (robrien) Date: 2008-09-14 01:30
or in FreeBSD?

2.6rc1 and 3.0b3 both fail test_array on FreeBSD7 amd64

test_array passes in 2.5.2 on the same machine but fails test_list the
same as test_array...

*** Signal 9
msg73209 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-09-14 10:56
2.6rc1 test_array passes on FreeBSD 6.3 i386.  I don't have a 7.0 box
(either i386 or amd64) accessible at the moment to cross check.

Can you run the test on its own in verbose mode to get a bit more of an
idea where its blowing up? - ie:

user@box$ ./python -E -tt Lib/test/regrtest.py -v test_array

from the build directory.
msg73225 - (view) Author: Reed O'Brien (robrien) Date: 2008-09-14 14:40
2.6rc2 and Python-3.0b3 test_array detail
 
test_alloc_overflow (test.test_array.DoubleTest) ... Killed

Fills swap space and dumps core.


2.5.2 

test_list
test_addmul (test.test_list.ListTest) ... ok
test_append (test.test_list.ListTest) ... ok
Killed

The FreeBSD ports patches fix this in 2.5.2. Specifically patching
seq_tests.py to limit test_bigrepeat() to  if sys.maxint <= 2147483647.
no other tests fail; so I don't know immediately what else is patched.
Although there are about 25 patches for the 2.5 port.
msg73256 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-09-15 12:15
I've temporarily got a 7.0 amd64 system running and can confirm what you
see.  I checked out the 2.5.2 port patches you mentioned, but all the
ones that seemed related are already in the 2.6rc1 sources.

On a hunch, I used ulimit -v is used to set a process virtual memory
size limit, and the test completed satisfactorily (I set it to 1048756,
ie 1GB).

I'm as yet none the wiser as to what to do with this info...
msg73257 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-09-15 12:44
On FreeBSD 7.0 i386, test_array passes without ulimit -v needing to be
set (ie its still the default "unlimited".
msg73299 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-09-16 13:10
Does the attached patch help fix this at all?  I encountered possibly the 
same problem on a 64-bit Mac build of Python, and this patch fixed the 
problem for me.
msg73335 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-09-17 16:15
Mark, your patch will probably get the test to pass, but the underlying
reason the test is failing appears to be unexpected behaviour of the
platform malloc().

FreeBSD 7.0 introduced a new malloc() implementation that relies on
mmap() and this is behaving differently to the malloc() implementation
in FreeBSD 6.3 which relied on sbrk().

I have posted a query about the new malloc()'s behaviour to a FreeBSD
forum and will report what I find out.
msg73367 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-09-18 08:23
Thanks, Alan.  I realised my answer was a shallow one after reading (too
late!) the thread you started on python-dev.

However, I have a suspicion that that particular array test
(test_alloc_overflow) was merely meant to test the code in
newarrayobject, at around line 427 of arraymodule.c, which looks like:

 	/* Check for overflow */
	if (nbytes / descr->itemsize != (size_t)size) {
		return PyErr_NoMemory();
	}

and that the test dated from an era when it was fairly safe to assume
that a size_t was at most 32 bits.  I'd guess that test_alloc_overflow
was never intended to be a test of OS malloc failure behaviour.

So the array test is wrong, and I think this patch should be applied
anyway.  I admit this doesn't help with the much more interesting
question of what's going on with malloc on FreeBSD.
msg73368 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-09-18 08:24
s/Alan/Andrew/.  Need more coffee.  Apologies.
msg73379 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-09-18 12:57
The patch looks fine to me, except that the failure message is now out
of date.
msg74076 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-09-30 13:35
On FreeBSD 7.0 amd64, applying Mark's patch does get the test to pass.

As noted on python-dev, the response I got from the author of the new
malloc() package in FreeBSD 7.x indicates that as of 7.1 the allocator
defaults to using sbrk() rather than mmap() (strategy selection is
tunable with an environment variable) so the problematic behaviour
should be masked once 7.1 is released.

Martin: I'm missing something regarding your comment about the failure
message - the exception text seems literally correct (though not
particularly useful), though I haven't spent a lot of time trying to
understand how test_array works.  What would you suggest?
msg74078 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2008-09-30 13:56
The 2**16 in the error message is wrong.  Maybe the error message should
be something like:

"Attempt to create array of size larger than maxsize failed to raise
MemoryError."

A bit verbose, but more accurate.

There's another error in that patch, too.  The line:

b * maxsize//3 + 1

should be

b * (maxsize//3 + 1)

(I was mistakenly reading the '*' as '*=', as in the first test.)
msg74096 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2008-09-30 19:48
> The 2**16 in the error message is wrong. 

As he says.
msg74113 - (view) Author: Andrew I MacIntyre (aimacintyre) * (Python triager) Date: 2008-10-01 03:30
Yes, the bleeding obvious became so after some sleep.

Committed to trunk as r66708 with the extra fix and a more abbreviated
failure message, which I hope is still semantically the same as Mark's
suggested text.

Will forward port as appropriate to py3k.
msg107979 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-06-17 01:18
Was the fix forward ported or is this an issue for 3.1/2?
msg185413 - (view) Author: Georg Brandl (georg.brandl) * (Python committer) Date: 2013-03-28 10:00
Closing this old pending issue due to lack of activity.
History
Date User Action Args
2013-03-28 10:00:00georg.brandlsetstatus: pending -> closed
nosy: + georg.brandl
messages: + msg185413

2010-06-17 01:18:05terry.reedysetstatus: open -> pending
nosy: + terry.reedy
messages: + msg107979

2008-10-01 03:30:45aimacintyresetmessages: + msg74113
2008-09-30 19:48:39loewissetmessages: + msg74096
2008-09-30 13:56:34mark.dickinsonsetmessages: + msg74078
2008-09-30 13:35:21aimacintyresetmessages: + msg74076
2008-09-18 12:57:11loewissetnosy: + loewis
messages: + msg73379
2008-09-18 08:24:23mark.dickinsonsetmessages: + msg73368
2008-09-18 08:23:24mark.dickinsonsetmessages: + msg73367
2008-09-17 16:15:26aimacintyresetmessages: + msg73335
2008-09-16 13:10:06mark.dickinsonsetfiles: + 64bit_2.patch
keywords: + patch
messages: + msg73299
nosy: + mark.dickinson
2008-09-15 12:44:21aimacintyresetmessages: + msg73257
2008-09-15 12:15:15aimacintyresetmessages: + msg73256
2008-09-14 14:40:15robriensetmessages: + msg73225
2008-09-14 10:56:58aimacintyresetnosy: + aimacintyre
messages: + msg73209
2008-09-14 01:30:18robriencreate