Author phillies
Recipients phillies
Date 2011-12-08.12:52:02
SpamBayes Score 5.10504e-07
Marked as misclassified No
Message-id <1323348723.51.0.524535293585.issue13555@psf.upfronthosting.co.za>
In-reply-to
Content
When I try to load a large file (>1GB) cPickle crashes with a MemoryError:
$python test.py
Traceback (most recent call last):
  File "/tmp/test.py", line 8, in <module>
    A2 = cPickle.load(f2)
MemoryError

test.py contains following code:
import numpy as np
import cPickle
A = np.random.randn(196,240000)
f = open('test.pydat', 'w')
cPickle.dump(A,f)
f.close()
f2 = open('test.pydat', 'rb')
A2 = cPickle.load(f2)

System:
cPickle 1.71
python 2.7.2
Ubuntu 11.10 amd64

Memory is not an issue as a) pickle works nicely and b) my computer has 122GB free RAM
History
Date User Action Args
2011-12-08 12:52:03philliessetrecipients: + phillies
2011-12-08 12:52:03philliessetmessageid: <1323348723.51.0.524535293585.issue13555@psf.upfronthosting.co.za>
2011-12-08 12:52:02phillieslinkissue13555 messages
2011-12-08 12:52:02philliescreate