Message253371
I very often want to use pickle to store huge objects, such that I do not need to recalculate them again.
However, I noticed that pickle uses O(n) (for n the object size in memory) amount of memory. That is, using python 3:
data = {'%06d' % i: i for i in range(30 * 1000 ** 2)}
# data consumes a lot of my 8GB ram
import pickle
with open('dict-database.p3', 'wb') as f: pickle.dump(data, f)
# I have to kill the process, in order to not overflow in memory. If I don't, the OS crashes. IMHO the OS should never crash due to python.
I don't think pickle should require a O(n) memory overhead. |
|
Date |
User |
Action |
Args |
2015-10-23 09:37:23 | prinsherbert | set | recipients:
+ prinsherbert |
2015-10-23 09:37:23 | prinsherbert | set | messageid: <1445593043.5.0.334623277706.issue25465@psf.upfronthosting.co.za> |
2015-10-23 09:37:23 | prinsherbert | link | issue25465 messages |
2015-10-23 09:37:22 | prinsherbert | create | |
|