I wrote a buggy PNG parser which ended up doing several file.read(large
value). It causes a MemoryError, which was strange because the file was
only a few KB long.

I tracked it down to the implementation of read(). When given a size
hint it preallocates the return string with that size. If the hint is
for 10MB then the string returned will be preallocated fro 10MB, even if
the actual read is empty.

Here's a reproducible

BLOCKSIZE = 10*1024*1024

f=open("empty.txt", "w")
f.close()

f=open("empty.txt")
data = []
for i in range(10000):
    s = f.read(BLOCKSIZE)
    assert len(s) == 0
    data.append(s)

I wasn't sure if this is properly a bug, but since the MemoryError
exception I got was quite unexpected and required digging into the
source code to figure out, I'll say that it is.

I can't reproduce, your code snippet works fine. What Python version is it?

I tested it with Python 2.5 on a Mac, Python 2.5 on FreeBSD, and Python
2.6b2+ (from SVN as of this morning) on a Mac.

Perhaps the memory allocator on your machine is making a promise it can't
keep?

Perhaps. I'm under Linux.

However, at the end of the file_read() implementation in fileobject.c,
you can find the following lines:

if (bytesread != buffersize)
_PyString_Resize(&v, bytesread);

Which means that the string *is* resized at the end.

You're right. I mistook the string implementation for the list one
which does keep a preallocated section in case of growth. Strings of
course don't grow so there's no need for that.

I tracked the memory allocation all the way down to
obmalloc.c:PyObject_Realloc . The call goes to realloc(p, nbytes) which
is a C lib call. It appears that the memory space is not reallocated.

That was enough to be able to find the python-dev thread "Darwin's
realloc(...) implementation never shrinks allocations" from Jan. 2005,
Bob Ippolito's post "realloc.. doesn’t?"
(http://bob.pythonmac.org/archives/2005/01/01/realloc-doesnt/ ) and bpo-1092502 .

Mind you, I also get the problem on FreeBSD 2.6 so it isn't Darwin
specific.

Le samedi 09 août 2008 à 11:26 +0000, Andrew Dalke a écrit :

Mind you, I also get the problem on FreeBSD 2.6 so it isn't Darwin
specific.

Darwin and the BSD's supposedly share a lot of common stuff.
But FreeBSD 2.6 is a bit old, isn't it?

FreeBSD is why my hosting provider uses. Freebsd.org calls 2.6 "legacy"
but the latest update was earlier this year.

There is shared history with Macs. I don't know the details though. I
just point out that the problem isn't only on Darwin.

FWIW:
I have performance problems on Windows XP (SP2) with Python 2.5.1 that
could be caused by this behaviour.
My code regularily calculates the sha1 sum of 10.000 files and because
in another reuse of the code had to deal with files too big to fit in memory
I set a limit of 256Mb. It looks like that is now allocated and
deallocated for every one of the 10.000 files, making things *very* slow.

My code regularily calculates the sha1 sum of 10.000 files and because
in another reuse of the code had to deal with files too big to fit in
memory I set a limit of 256Mb.

Why don't you use a sensible buffer size, e.g. 1MB? Reading data in
256MB chunks sounds foolish in any case.

Andrew, as for memory reallocation issues, you may take a look at bpo-3526
where someone has similar problems on SunOS.

If nobody objects, I will close the present bug as invalid.

Le jeudi 11 septembre 2008 à 16:01 +0200, Anthon van der Neut a écrit :

The thing however was resolved by reading multiple smaller chunks indeed
1Mb if the filesize exceeds 1Mb (in the latter case the original read()
is done.

It's too complicated. Just use chunks in all cases (even small files)
and you are done. There should be no visible performance downside to
doing so.

Using fixed-size chunks to read binary data from a file of an unknown
size isn't a Python-specific idiom, really. It's the same in C, C++,
PHP, etc.

I'm still undecided on if this is a bug or not. The problem occurs even
when I'm not reading "data from a file of an unknown size." My example
causes a MemoryError on my machine even though the file I'm reading
contains 0 bytes.

The problem is Python's implementation is "alloc the requested bytes and
truncate if needed" vs what I expected "read chunks at a time up to the
requested number of bytes." There's nothing in the documentation which
states the implementation, although "Note that this method may call the
underlying C function fread more than once in an effort to acquire as
close to size bytes as possible." leans slightly towards my
interpretation.

I looked a little for real-world cases that could cause a denial-of-
service attack but didn't find one.

If there is a problem, it will occur very rarely. Go ahead an mark it
as "will not fix" or something similar. I don't think the change in the
code is justifiable.