When I try to load a large file (>1GB) cPickle crashes with a MemoryError:
$python test.py
Traceback (most recent call last):
  File "/tmp/test.py", line 8, in <module>
    A2 = cPickle.load(f2)
MemoryError

test.py contains following code:
import numpy as np
import cPickle
A = np.random.randn(196,240000)
f = open('test.pydat', 'w')
cPickle.dump(A,f)
f.close()
f2 = open('test.pydat', 'rb')
A2 = cPickle.load(f2)

System:
cPickle 1.71
python 2.7.2
Ubuntu 11.10 amd64

Memory is not an issue as a) pickle works nicely and b) my computer has 122GB free RAM

Maybe Ubuntu doesn't think it is safe to allocate the memory.

Well, replace cPickle by pickle and it works. So if there is a memory allocation problem cPickle should be able to handle it, especially since it should be completely compatible to pickle.

Have you checked the system monitor after all cPickle can use more memory than 1GB.

Is there a way to reproduce that doesn't involve numpy?

> my computer has 122GB free RAM

122, or 1.22?

> Well, replace cPickle by pickle and it works.

cPickle makes some direct call to malloc()/realloc()/free(), contrarily to pickle which uses pymalloc. This could lead to heap fragmentation.

What does the following return ?
$ /usr/bin/time -v python test.py

a) it's 122GB free RAM (out of 128GB total RAM)

b) when I convert the numpy array to a list it works. So seems to be a problem with cPickle and numpy at/from a certain array size

c) $ /usr/bin/time -v python test_np.py 
Traceback (most recent call last):
  File "test_np.py", line 12, in <module>
    A2 = cPickle.load(f2)
MemoryError
Command exited with non-zero status 1
	Command being timed: "python test_np.py"
	User time (seconds): 73.72
	System time (seconds): 4.56
	Percent of CPU this job got: 87%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:29.52
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 7402448
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 0
	Minor (reclaiming a frame) page faults: 726827
	Voluntary context switches: 41043
	Involuntary context switches: 7793
	Swaps: 0
	File system inputs: 3368
	File system outputs: 2180744
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 1

hth

@Antoine
Couldn't this be linked to #11564 (pickle not 64-bit ready)? AFAICT this wasn't fixed in 2.7. Basically, an integer overflow, and malloc() would bail out when asked a ridiculous size.

@Philipp
I'd be curious to see the last lines of
$ ltrace -e malloc python test_np.py

you'll probably see a call to malloc() return NULL.

> Couldn't this be linked to #11564 (pickle not 64-bit ready)?

Well, I don't know anything about numpy, but:

>>> 196 * 240000
47040000
>>> 196 * 240000 * 8   # assuming 8 bytes per float
376320000
>>> 2**31
2147483648

So it seems unlikely to be the explanation. After all the report says ">1GB" for the file size, which is still comfortably in the capabilities of a 32-bit module.

Philipp, perhaps you could try to run Python under gdb and try to diagnose where the MemoryError occurs?

Also, posting the disassembly (using pickletools.dis()) of a very small equivalent pickle (e.g. of randn(2,3)) could help us know what is involved.

Oh, what's more interesting is that it works here (Python 2.7.1 and numpy 1.6.1 under Mageia with 8GB RAM). Looking at a pickle disassembly, the only remarkable thing is the presence of a long binary string (the raw serialization of all IEEE floats), which shouldn't give any problem to cPickle (and indeed doesn't). What is your numpy version, Philipp?

> So it seems unlikely to be the explanation.

Victor reproduced in on IRC, and it's indeed an overflow.
The problematic code is in readline_file:
"""
        bigger = self->buf_size << 1;
        if (bigger <= 0) {              /* overflow */
            PyErr_NoMemory();
            return -1;
        }
        newbuf = (char *)realloc(self->buf, bigger);
        if (!newbuf)  {
            PyErr_NoMemory();
            return -1;
        }
"""

self->buf_size is an int, which overflow pretty easily.

>>> 196 * 240000
47040000
>>> 196 * 240000 * 8   # assuming 8 bytes per float
376320000
>>> 2**31
2147483648

Hmmm... A byte is 8 bit, which gives:
>>> 196 * 240000 * 8 * 8
3010560000L
>>> 196 * 240000 * 8 * 8 > 2**31
True

Now, if it works on your box, it's probably due to the compiler optimizing the check away. Since `bigger` is cast to an unsigned 64-bit (size_t) when calling malloc(), it happens to work.
Maybe your distro doesn't build python with -fwrapv.

So, what do you suggest? Should we fix this (Py_ssize_t, overflow check before computation), as in #11564?

> Should we fix this (Py_ssize_t, overflow check before computation), as in #11564?

Yes. Use Py_ssize_t type for the buf_size attribute, and replace "bigger <= 0" (test if an overflow occurred) by "self->buf_size > (PY_SSIZE_T_MAX >> 1)".

Ah, I see. It's a bit of a pity not to be able to load files > 1GB, especially on a 64-bit build (!). Perhaps cPickle could be made partly 64-bit compatible? Or at least, indeed, do a proper anti-overflow check.

Here's a patch which should fix this. However, I'm unable to test it.

I think there's a problem here:

+    self->data = realloc(self->data, self->size * sizeof(PyObject *));
+    if (self->data == NULL)
         goto nomemory;

If realloc() fails, the old data pointer is lost and therefore will never get free()ed.

Same for:

+        self->buf = (char *)realloc(self->buf, self->buf_size);

Here:

-        int *marks;
-        s=self->marks_size+20;
-        if (s <= self->num_marks) s=self->num_marks + 1;
+        size_t alloc;
+        Py_ssize_t *marks;
+
+        /* Use the size_t type to check for overflow. */
+        alloc = ((size_t)self->num_marks << 1) + 20;

It seems you are changing the overallocation algorithm (from additive to multiplicative). I'm not sure it should be in the scope of the patch, although multiplicative overallocation is generally better.

New version.

> New version.

Looks good to me.

Here's a new version with a test (untested).
Note that I'm absolutely not sure that the 'memsize' argument to
bigmemtest is correct.

Antoine, could you test the last version (test_pickle and if possible
with the OP testcase)?
I can't test it myself (32-bit machine with 1 GB).

Le vendredi 06 janvier 2012 à 19:17 +0000, Charles-François Natali a
écrit :
> Charles-François Natali <neologix@free.fr> added the comment:
> 
> Antoine, could you test the last version (test_pickle and if possible
> with the OP testcase)?
> I can't test it myself (32-bit machine with 1 GB).

Well, first, there are compilation warnings on a 64-bit box:

gcc -pthread -fPIC -fno-strict-aliasing -g -O2 -g -O0 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I/usr/local/include -I/home/antoine/cpython/27/Include -I/home/antoine/cpython/27 -c /home/antoine/cpython/27/Modules/cPickle.c -o build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o
/home/antoine/cpython/27/Modules/cPickle.c: In function ‘put2’:
/home/antoine/cpython/27/Modules/cPickle.c:810:9: attention : format ‘%d’ expects type ‘int’, but argument 4 has type ‘Py_ssize_t’
/home/antoine/cpython/27/Modules/cPickle.c: In function ‘load_mark’:
/home/antoine/cpython/27/Modules/cPickle.c:4610:21: attention : assignment from incompatible pointer type
gcc -pthread -shared build/temp.linux-x86_64-2.7-pydebug/home/antoine/cpython/27/Modules/cPickle.o -L/usr/local/lib -o build/lib.linux-x86_64-2.7-pydebug/cPickle.so

Second, I can't seem to get the test to work with 8GB RAM (approximately
6.5GB free according to "free"). The MemoryError is quite expectable for
test_pickle, though, since the code there doesn't try to conserve memory
at all:

test test_pickle failed -- Traceback (most recent call last):
  File "/home/antoine/cpython/27/Lib/test/test_support.py", line 983, in wrapper
    return f(self, maxsize)
  File "/home/antoine/cpython/27/Lib/test/pickletester.py", line 1298, in test_huge_str_32b
    pickled = self.dumps(data, proto)
  File "/home/antoine/cpython/27/Lib/test/test_pickle.py", line 74, in dumps
    return pickle.dumps(arg, proto)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 1374, in dumps
    Pickler(file, protocol).dump(obj)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 224, in dump
    self.save(obj)
  File "/home/antoine/cpython/27/Lib/pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "/home/antoine/cpython/27/Lib/pickle.py", line 488, in save_string
    self.write(STRING + repr(obj) + '\n')
MemoryError

I would therefore suggest to enable the test only for cPickle.
For test_cpickle the behaviour is different:
- for protocol 0, I get a MemoryError (which may be expected, if the
test truly needs more than 6.5GB, for example because of suboptimal
buffer management)
- for protocol 1 and 2, I get "SystemError: error return without
exception set"

Could someone with a 64-bit box take over this one?
I won't go anywhere with my 32-bit box...

Bump.
@neologix
I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap)

I may tackle this but rare 2.7-only bugs are pretty low on my priorities list.

> I have a 64-bit laptop with 2 GB memory so I don't think I can do so. (though one could use swap)

AFAICT, a binary string a little longer than 1GB should be enough to
reproduce the bug.
Just make sure Python isn't built with '-fwrapv'.

Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments.

> Charles-François, are you going to finish this before 2.7.4 RC released? The patch should be updated to address Antoine's comments.

No.
I don't have access to a 64-bit box, which makes it difficult to write
(I don't get compiler warnings) and test.

Feel free to pick up the patch and update it!

I have picked up it.

Actually BINSTRING format supports only strings less 2GiB, so test should be changed. But there are other bugs which prohibit pickling a lot of data. For one I open issue17054.

Here is an updated patch. Fixed two bugs found by Antoine (an inappropriate format and a memory error in bigmemtest), fixed resizing of marks array and one possible integer overflow in write_other(). Used workaround to bypass limitations of cStringIO API.

Test updated too. Now it doesn't try to write a string larger than 2 GiB (it's impossible), instead writes a lot of shorter strings with total size larger than 2 GiB.

New changeset 680959a3ae2e by Serhiy Storchaka in branch '2.7':
Issue #13555: cPickle now supports files larger than 2 GiB.
http://hg.python.org/cpython/rev/680959a3ae2e

680959a3ae2e has caused that g-ir-scanner tool from gobject-introspection (which seems to create some pickles and next load them?) fails with MemoryError, e.g. during building of GTK+.
MemoryError occurs only for a subset of pickles.
I attach a small pickle, which allows to reproduce this problem.
Download this pickle and save it in /tmp.

Additional preparation:
$ cd /tmp
$ wget -q http://ftp.gnome.org/pub/gnome/sources/gobject-introspection/1.34/gobject-introspection-1.34.2.tar.xz
$ tar -xJf gobject-introspection-1.34.2.tar.xz
$ sed -e '/^with LibtoolImporter/,/^$/d' -i gobject-introspection-1.34.2/giscanner/xmlwriter.py

Behavior before 680959a3ae2e:
$ PYTHONPATH="gobject-introspection-1.34.2" python2.7 -c 'import cPickle; print(cPickle.load(open("pickle")))'
<giscanner.girparser.GIRParser object at 0x7f506299be90>

Behavior after 680959a3ae2e:
$ PYTHONPATH="gobject-introspection-1.34.2" python2.7 -c 'import cPickle; print(cPickle.load(open("pickle")))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
MemoryError

New changeset f3f23ecdb1c6 by Serhiy Storchaka in branch '2.7':
Issue #13555: Fix an integer overflow check.
http://hg.python.org/cpython/rev/f3f23ecdb1c6

Thank you for the report. Standard tests do not cover pickling/unpickling to real files.

I have opened issue17299 for testing issue.