This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

Title: list(xrange(sys.maxint / 4)) -> swapping
Type: Stage:
Components: Interpreter Core Versions: Python 2.3
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: brianl, loewis, nnorwitz
Priority: normal Keywords:

Created on 2002-09-29 02:00 by brianl, last changed 2022-04-10 16:05 by admin. This issue is now closed.

Messages (7)
msg12538 - (view) Author: Brian Lenihan (brianl) Date: 2002-09-29 02:00
Refer to #556025 list(xrange(1e9)) --> seg fault for
the history.  The test in test/ causes OS X
10.2.1 to start swapping instantly and never stopped
until I killed the process after 5 minutes.
I wasn't brave enough to let it run for any longer.

list(xrange(1e09)) or even list(xrange(sys.maxint)) works:

Traceback (most recent call last):
  File "", line 553, in ?

PhysMem:  55.7M wired,  300M active,  150M inactive, 
506M used, 6.19M free
VM: 4.19G + 76.9M   36909(35) pageins, 1505603(200)

 1157 python       9.1%  0:33.72   1    13  1186  
373M+ 1.55M   160M- 2.29G+
msg12539 - (view) Author: Martin v. Löwis (loewis) * (Python committer) Date: 2002-09-30 16:01
Logged In: YES 

Can you please find out how much memory it tries to
allocate? The easiest approach might be add a print
statement; I *think* the relevant allocation should come
from the NRESIZE macro - we'd need to find out what value
_new_size has.
msg12540 - (view) Author: Brian Lenihan (brianl) Date: 2002-10-01 04:30
Logged In: YES 

OK, I added a fprintf to the NRESIZE macro  and got the same
results on three different platforms, but both OS X builds
start swapping rather than raising a MemoryError.  I guess
the next place to look is PyMem_RESIZE

Linux (glibc2.1.3):
OS X 10.2.1 (gcc3):
OS X 10.2.1 (gcc 2.95.2):

sys.maxint: 2147483647
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 8
_new_size: 536870912

msg12541 - (view) Author: Brian Lenihan (brianl) Date: 2002-10-06 09:31
Logged In: YES 

I did a little more digging on this problem.  I don't
understand why this is happening yet, but I also think I
found a bug in obmalloc.c

I built a debug interpreter with PYMALLOC_DEBUG after I ran in gdb and saw memset being called with a bogus

#1  0x0009722c in _PyObject_DebugMalloc (nbytes=2147483648)
at Objects/obmalloc.c:998

nbytes here should be sys.maxint / 4 + 16, but instead
memset is being called with sys.maxint + 16.

obmalloc.c line 981:

total = nbytes + 16
if (total < nbytes || (total >> 31) > 1) {
  return NULL

total is a uchar and nbytes is a size_t.  Neither side of
this test is true if total = sys.maxint + 16 has overflowed:
total: 2147483664 nbytes: -2147483632

It seems to me the test should be:

if (total < nbytes || (total >> 31) >= 1) {

Changing obmalloc.c as above, fixes the swapping, but brings
me no closer to finding the actual cause of the poblem.
msg12542 - (view) Author: Brian Lenihan (brianl) Date: 2002-10-07 22:29
Logged In: YES 

Forget what I said in the last comment.  I must have been

The problem boils down to this:  OS X does lazy memory
allocation and each process gets up to 4GB of virtual memory.

I now know a lot more about how the interpreter manages
memory.  I could have saved myself the trouble by paying
attention to what the VM: 4.19G+ in top was telling me.

The test in should not be run on OS X.  I am
going to avoid problems like this in the future by setting a
hard limit on per process memory of 400MB.
msg12543 - (view) Author: Neal Norwitz (nnorwitz) * (Python committer) Date: 2002-10-14 20:49
Logged In: YES 

This problem was recently addressed by changing sys.maxint /
4 to sys.maxint / 2.  Brian, do you still have the problem?
msg12544 - (view) Author: Brian Lenihan (brianl) Date: 2002-10-14 21:12
Logged In: YES 

Nope.  Tim's change fixed it.  I've verified that it works
in 2.3 on OS X 10.2.1 and 2.2.2 on XP.
Date User Action Args
2022-04-10 16:05:42adminsetgithub: 37235
2002-09-29 02:00:35brianlcreate