I'm using the standard lib SimpleXMLRPCServer to receive a file from a
client. The file is sent using an instance of the xmlrpclib.Binary class. 

Whenever the file is bigger than a few MB, the server reports spews out
an internal error to stderr:

Python(13950) malloc: *** mmap(size=9916416) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

This *only* happens if the client and server are not the same machine;
i.e., connecting and sending to "localhost" always works.

Attached testcase:

* test_server.py opens a server on port 8001 waiting to receive a blob
of data over HTTP.

* Modify test_client.py to point at a big file, and then connect to the
server machine, and you will ses the error.


I'm running OS X-bundled Python 2.5.1 on OS X 10.5.2

test_server.py and test_client.py zipped attached.

First and foremost:  do not use XML for bulk data transport.  It is
HORRIBLY inefficient.

I've been playing with this on Linux and OS X with various (trunk 2.6,
release25-maint and 2.5.2) pythons:

I was never able to reproduce the malloc failures on my systems, testing
with data sizes up to 100mb.  It likely takes a specific set of
conditions to reproduce exactly that problem but I do understand how it
could happen.

Anyways one -likely- source of such problems was the socket module
_fileobject.recv() code's long lived over-allocated+realloced strings. 
This was "fixed" in release25-maint [to become 2.5.3] (actually it
caused a perf regression in other code) and the fix was fixed to solve
the perf regression in trunk and will be backported...  Too much history
to sum up there.  See http://bugs.python.org/issue2632 and the older
issues it links to for details.

I cannot claim that the above solves this problem because the bulk of
the actual memory used is the XML parser's fault:

Instrumenting the SimpleXMLRPCServer do_POST code I see the following:

The majority of the memory bloat to handle a request (bloat appears to
be 5-10x the size of the Binary data blob in question!) comes from the
XML parser called by xmlrpclib.loads() from SimpleXMLRPCServer's
_marshaled_dispatch() method.

Why?  Its XML.  On top of that it is not being parsed and decoded as a
stream.

hwaara: can you try your test code on the systems where you can
reproduce this problem using the most recent Python 2.6 alpha release to
see if the mmap errors still happen?

I'll try to test this.

Are there pre-built binaries for 2.6 alpha, or will I need to compile it
myself?

There are no prebuilt binaries for OS X, just download the latest source
tarball from here and build it:  http://python.org/download/releases/2.6/

Thanks for testing.

Please test on 2.6 or better, 2.7 (rc already out) and 3.1 to verify there still is a problem.

I'm not sure whether it's related but on psutil we have a similar error message by using Python 2.6.1 64-bit:
http://code.google.com/p/psutil/issues/detail?id=135

This still needs to be verified that there is a problem with a *current* release. That means 2.7.1 (rc just out) or 3.2 (a4 just out).

If this would be considered a security issue (I do not know) then 'current' includes 2.5.5, 2.6.6, and 3.1.3 (rc or final).

Without such verification, this should be closed until the problem appears again.

I can try to do some testing to reproduce w/ 2.7: 2.5 was IIRC 32-bit on leopard by default though, so should I force a non-64-bit build to test this? I'm not entirely sure if that'll change things, but want to make sure. I can test with both 2.5 and 2.7 on leopard.

Never verified for current release. 2.5 closed.