classification
Title: "error: can't allocate region" from mmap() when receiving big chunk of data
Type: crash Stage:
Components: Versions: Python 2.5
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: giampaolo.rodola, gregory.p.smith, hwaara, ixokai, terry.reedy
Priority: normal Keywords:

Created on 2008-05-18 00:16 by hwaara, last changed 2011-06-01 06:24 by terry.reedy. This issue is now closed.

Files
File name Uploaded Description Edit
testcase.zip hwaara, 2008-05-18 00:18 test_server.py and test_client.py (zipped)
Messages (11)
msg67018 - (view) Author: Håkan Waara (hwaara) Date: 2008-05-18 00:16
I'm using the standard lib SimpleXMLRPCServer to receive a file from a
client. The file is sent using an instance of the xmlrpclib.Binary class. 

Whenever the file is bigger than a few MB, the server reports spews out
an internal error to stderr:

Python(13950) malloc: *** mmap(size=9916416) failed (error code=12)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug

This *only* happens if the client and server are not the same machine;
i.e., connecting and sending to "localhost" always works.

Attached testcase:

* test_server.py opens a server on port 8001 waiting to receive a blob
of data over HTTP.

* Modify test_client.py to point at a big file, and then connect to the
server machine, and you will ses the error.


I'm running OS X-bundled Python 2.5.1 on OS X 10.5.2
msg67019 - (view) Author: Håkan Waara (hwaara) Date: 2008-05-18 00:18
test_server.py and test_client.py zipped attached.
msg67401 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2008-05-27 00:10
First and foremost:  do not use XML for bulk data transport.  It is
HORRIBLY inefficient.

I've been playing with this on Linux and OS X with various (trunk 2.6,
release25-maint and 2.5.2) pythons:

I was never able to reproduce the malloc failures on my systems, testing
with data sizes up to 100mb.  It likely takes a specific set of
conditions to reproduce exactly that problem but I do understand how it
could happen.

Anyways one -likely- source of such problems was the socket module
_fileobject.recv() code's long lived over-allocated+realloced strings. 
This was "fixed" in release25-maint [to become 2.5.3] (actually it
caused a perf regression in other code) and the fix was fixed to solve
the perf regression in trunk and will be backported...  Too much history
to sum up there.  See http://bugs.python.org/issue2632 and the older
issues it links to for details.

I cannot claim that the above solves this problem because the bulk of
the actual memory used is the XML parser's fault:

Instrumenting the SimpleXMLRPCServer do_POST code I see the following:

The majority of the memory bloat to handle a request (bloat appears to
be 5-10x the size of the Binary data blob in question!) comes from the
XML parser called by xmlrpclib.loads() from SimpleXMLRPCServer's
_marshaled_dispatch() method.

Why?  Its XML.  On top of that it is not being parsed and decoded as a
stream.
msg67402 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2008-05-27 00:11
hwaara: can you try your test code on the systems where you can
reproduce this problem using the most recent Python 2.6 alpha release to
see if the mmap errors still happen?
msg67405 - (view) Author: Håkan Waara (hwaara) Date: 2008-05-27 08:06
I'll try to test this.

Are there pre-built binaries for 2.6 alpha, or will I need to compile it
myself?
msg67413 - (view) Author: Gregory P. Smith (gregory.p.smith) * (Python committer) Date: 2008-05-27 16:32
There are no prebuilt binaries for OS X, just download the latest source
tarball from here and build it:  http://python.org/download/releases/2.6/

Thanks for testing.
msg107428 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-06-09 22:23
Please test on 2.6 or better, 2.7 (rc already out) and 3.1 to verify there still is a problem.
msg121307 - (view) Author: Giampaolo Rodola' (giampaolo.rodola) * (Python committer) Date: 2010-11-16 17:58
I'm not sure whether it's related but on psutil we have a similar error message by using Python 2.6.1 64-bit:
http://code.google.com/p/psutil/issues/detail?id=135
msg121309 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2010-11-16 18:45
This still needs to be verified that there is a problem with a *current* release. That means 2.7.1 (rc just out) or 3.2 (a4 just out).

If this would be considered a security issue (I do not know) then 'current' includes 2.5.5, 2.6.6, and 3.1.3 (rc or final).

Without such verification, this should be closed until the problem appears again.
msg121313 - (view) Author: Stephen Hansen (ixokai) Date: 2010-11-16 20:08
I can try to do some testing to reproduce w/ 2.7: 2.5 was IIRC 32-bit on leopard by default though, so should I force a non-64-bit build to test this? I'm not entirely sure if that'll change things, but want to make sure. I can test with both 2.5 and 2.7 on leopard.
msg137420 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2011-06-01 06:24
Never verified for current release. 2.5 closed.
History
Date User Action Args
2011-06-01 06:24:37terry.reedysetstatus: open -> closed
resolution: out of date
messages: + msg137420
2010-11-16 20:08:08ixokaisetstatus: pending -> open

messages: + msg121313
2010-11-16 18:45:23terry.reedysetstatus: open -> pending

messages: + msg121309
2010-11-16 17:59:32giampaolo.rodolasetnosy: + ixokai
2010-11-16 17:58:59giampaolo.rodolasetstatus: pending -> open
nosy: + giampaolo.rodola
messages: + msg121307

2010-06-09 22:23:35terry.reedysetstatus: open -> pending
nosy: + terry.reedy
messages: + msg107428

2008-05-27 16:32:00gregory.p.smithsetmessages: + msg67413
2008-05-27 08:06:51hwaarasetmessages: + msg67405
2008-05-27 00:11:09gregory.p.smithsetmessages: + msg67402
2008-05-27 00:10:05gregory.p.smithsetmessages: + msg67401
2008-05-25 08:57:10gregory.p.smithsetpriority: normal
nosy: + gregory.p.smith
2008-05-18 00:22:51benjamin.petersonsettype: crash
2008-05-18 00:18:47hwaarasetfiles: + testcase.zip
messages: + msg67019
2008-05-18 00:16:53hwaaracreate