classification
Title: 2 GB limit in array module
Type: behavior Stage:
Components: Extension Modules Versions: Python 3.1, Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: mark.dickinson Nosy List: mark.dickinson, mcfrith, skrah
Priority: normal Keywords:

Created on 2010-08-06 02:20 by mcfrith, last changed 2010-08-10 18:31 by mark.dickinson. This issue is now closed.

Messages (4)
msg113061 - (view) Author: Martin Frith (mcfrith) Date: 2010-08-06 02:20
The array module does not seem to work for arrays > 2 GB.

>>> import sys, array
>>> sys.maxsize
9223372036854775807
>>> x = array.array('c', ('a' for i in xrange(300000000)))
(seems to work OK)
>>> x = array.array('c', ('a' for i in xrange(3000000000)))
(runs forever: memory usage increases to 2GB and then stops)

I think the cause is: casting to int in arraymodule.c, in array_iter_extend() and array_append().

My uname -a:
Linux c016 2.6.18-164.el5 #1 SMP Thu Sep 3 03:28:30 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
msg113088 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-06 09:47
Thanks for the report.  Also applies to 3.1 and 3.2 (and 2.6, but it's too late to fix things there).  I agree that those (int) casts look wrong---I suspect they're leftovers from the big int -> Py_ssize_t switch that happened for Python 2.5.

I've removed the casts in svn revisions 83751 through 83754.  Unfortunately, right now I don't have access to a machine with enough memory to test that this fixes the problem.  Martin, would you be able to check this?  You'll need a Python svn checkout if you don't already have one;  instructions for getting it are at http://python.org/dev
msg113349 - (view) Author: Stefan Krah (skrah) * (Python committer) Date: 2010-08-08 23:14
Tested on an 8GB machine. This is fixed.
msg113554 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-10 18:31
Thanks for confirming, Stefan.
History
Date User Action Args
2010-08-10 18:31:27mark.dickinsonsetmessages: + msg113554
2010-08-09 02:28:59benjamin.petersonsetstatus: pending -> closed
2010-08-08 23:14:15skrahsetstatus: open -> pending
2010-08-08 23:14:07skrahsetstatus: pending -> open
nosy: + skrah
messages: + msg113349

2010-08-06 09:47:45mark.dickinsonsetstatus: open -> pending

assignee: mark.dickinson
versions: + Python 3.1, Python 3.2
nosy: + mark.dickinson

messages: + msg113088
resolution: fixed
2010-08-06 02:20:52mcfrithcreate