Issue616019
This issue tracker has been migrated to GitHub,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2002-09-29 02:00 by brianl, last changed 2022-04-10 16:05 by admin. This issue is now closed.
Messages (7) | |||
---|---|---|---|
msg12538 - (view) | Author: Brian Lenihan (brianl) | Date: 2002-09-29 02:00 | |
Refer to #556025 list(xrange(1e9)) --> seg fault for the history. The test in test/test_b1.py causes OS X 10.2.1 to start swapping instantly and never stopped until I killed the process after 5 minutes. I wasn't brave enough to let it run for any longer. list(xrange(1e09)) or even list(xrange(sys.maxint)) works: Traceback (most recent call last): File "test_b1.py", line 553, in ? list(xrange(sys.maxint)) MemoryError PhysMem: 55.7M wired, 300M active, 150M inactive, 506M used, 6.19M free VM: 4.19G + 76.9M 36909(35) pageins, 1505603(200) pageouts PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT RSHRD RSIZE VSIZE 1157 python 9.1% 0:33.72 1 13 1186 373M+ 1.55M 160M- 2.29G+ |
|||
msg12539 - (view) | Author: Martin v. Löwis (loewis) * | Date: 2002-09-30 16:01 | |
Logged In: YES user_id=21627 Can you please find out how much memory it tries to allocate? The easiest approach might be add a print statement; I *think* the relevant allocation should come from the NRESIZE macro - we'd need to find out what value _new_size has. |
|||
msg12540 - (view) | Author: Brian Lenihan (brianl) | Date: 2002-10-01 04:30 | |
Logged In: YES user_id=17671 OK, I added a fprintf to the NRESIZE macro and got the same results on three different platforms, but both OS X builds start swapping rather than raising a MemoryError. I guess the next place to look is PyMem_RESIZE Windows-XP-5.1.2600-SP1: Linux (glibc2.1.3): OS X 10.2.1 (gcc3): OS X 10.2.1 (gcc 2.95.2): sys.maxint: 2147483647 test_b1.py: list _new_size: 8 _new_size: 8 _new_size: 8 _new_size: 8 _new_size: 8 _new_size: 536870912 |
|||
msg12541 - (view) | Author: Brian Lenihan (brianl) | Date: 2002-10-06 09:31 | |
Logged In: YES user_id=17671 I did a little more digging on this problem. I don't understand why this is happening yet, but I also think I found a bug in obmalloc.c I built a debug interpreter with PYMALLOC_DEBUG after I ran test_b1.py in gdb and saw memset being called with a bogus argument: #1 0x0009722c in _PyObject_DebugMalloc (nbytes=2147483648) at Objects/obmalloc.c:998 nbytes here should be sys.maxint / 4 + 16, but instead memset is being called with sys.maxint + 16. obmalloc.c line 981: total = nbytes + 16 if (total < nbytes || (total >> 31) > 1) { .... return NULL total is a uchar and nbytes is a size_t. Neither side of this test is true if total = sys.maxint + 16 has overflowed: total: 2147483664 nbytes: -2147483632 It seems to me the test should be: if (total < nbytes || (total >> 31) >= 1) { Changing obmalloc.c as above, fixes the swapping, but brings me no closer to finding the actual cause of the poblem. |
|||
msg12542 - (view) | Author: Brian Lenihan (brianl) | Date: 2002-10-07 22:29 | |
Logged In: YES user_id=17671 Forget what I said in the last comment. I must have been asleep. The problem boils down to this: OS X does lazy memory allocation and each process gets up to 4GB of virtual memory. I now know a lot more about how the interpreter manages memory. I could have saved myself the trouble by paying attention to what the VM: 4.19G+ in top was telling me. The test in test_b1.py should not be run on OS X. I am going to avoid problems like this in the future by setting a hard limit on per process memory of 400MB. |
|||
msg12543 - (view) | Author: Neal Norwitz (nnorwitz) * | Date: 2002-10-14 20:49 | |
Logged In: YES user_id=33168 This problem was recently addressed by changing sys.maxint / 4 to sys.maxint / 2. Brian, do you still have the problem? |
|||
msg12544 - (view) | Author: Brian Lenihan (brianl) | Date: 2002-10-14 21:12 | |
Logged In: YES user_id=17671 Nope. Tim's change fixed it. I've verified that it works in 2.3 on OS X 10.2.1 and 2.2.2 on XP. |
History | |||
---|---|---|---|
Date | User | Action | Args |
2022-04-10 16:05:42 | admin | set | github: 37235 |
2002-09-29 02:00:35 | brianl | create |