classification
Title: Failure to create multiprocessing shared arrays larger than 50% of memory size under linux
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: mboquien, neologix, pitrou, python-dev, sbt, serhiy.storchaka
Priority: normal Keywords: patch

Created on 2014-03-31 19:37 by mboquien, last changed 2015-04-14 20:58 by pitrou. This issue is now closed.

Files
File name Uploaded Description Edit
shared_array.diff mboquien, 2014-03-31 19:55 review
shared_array.diff mboquien, 2014-03-31 20:24 review
Messages (18)
msg215258 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-03-31 19:37
It is currently impossible to create multiprocessing shared arrays larger than 50% of memory size under linux (and I assume other unices). A simple test case would be the following:

from multiprocessing.sharedctypes import RawArray
import ctypes

foo = RawArray(ctypes.c_double, 10*1024**3//8)  # Allocate 10GB array

If the array is larger than 50% of the total memory size, the process get SIGKILL'ed by the OS. Deactivate the swap for better effects.

Naturally this requires that the tmpfs max size is large enough, which is the case here, 15GB max with 16GB of RAM.

I have tracked down the problem to multiprocessing/heap.py. The guilty line is: f.write(b'\0'*size). Indeed, for very large sizes it is going to create a large intermediate array (10 GB in my test case) and as much memory is going to be allocated to the new shared array, leading to a memory consumption over the limit.

To solve the problem, I have split the zeroing of the shared array into blocks of 1MB. I can now allocate arrays as large as the tmpfs maximum size. Also it runs a bit faster. On a test case of a 6GB RawArray, 3.4.0 takes a total time of 3.930s whereas it goes down to 3.061s with the attached patch.
msg215260 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-03-31 19:55
Updated the patch not to create a uselessly large array if the size is small than the block size.
msg215264 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-03-31 20:24
New update of the patch following Antoine Pitrou's comments. PEP8 does not complain anymore.
msg215268 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-03-31 21:18
You overlooked the part where I was suggesting to add a unit test :-)
Also, you'll have to sign a contributor's agreement at https://www.python.org/psf/contrib/contrib-form/

Thanks!
msg215296 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-04-01 07:03
I have now signed the contributor's agreement.

As for the unit test I was looking at it. However, I was wondering how to write a test that would have triggered the problem. It only shows up for very large arrays and it depends on occupied memory and the configuration of the temp dir. Or should I simply write a test creating for instance a 100 MB array and checking it has the right length?
msg215404 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-02 21:20
Zero-filling mmap's backing file isn't really optimal: why not use truncate() instead? This way, it'll avoid completely I/O on filesystems that support sparse files, and should still work on FS that don't.
msg215407 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-04-02 22:13
If I remember correctly the problem is that some OS like linux (and probably others) do not really allocate space until something is written. If that's the case then the process may get killed later on when it writes something in the array.

Here is a quick example:

$ truncate -s 1T test.file
$ ls -lh test.file 
-rw-r--r-- 1 mederic users 1.0T Apr  2 23:10 test.file
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1       110G   46G   59G  44% /home
msg215408 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2014-04-02 23:20
Using truncate() to zero extend is not really portable: it is only guaranteed on XSI-compliant POSIX systems.

Also, the FreeBSD man page for mmap() has the following warning:

WARNING! Extending a file with ftruncate(2), thus creating a big
hole, and then filling the hole by modifying a shared mmap() can
lead to severe file fragmentation.  In order to avoid such
fragmentation you should always pre-allocate the file's backing
store by write()ing zero's into the newly extended area prior to
modifying the area via your mmap().  The fragmentation problem is
especially sensitive to MAP_NOSYNC pages, because pages may be
flushed to disk in a totally random order.
msg215425 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-03 06:14
> If I remember correctly the problem is that some OS like linux (and
probably others) do not really allocate space until something is written.
If that's the case then the process may get killed later on when it writes
something in the array.

Yes, it's called overcommitting, and it's a good thing. It's exactly the
same thing for memory: malloc() can return non-NULL, and the process will
get killed when first writing to the page in case of memory pressure.
msg215433 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-04-03 09:01
"the process will get killed when first writing to the page in case of memory pressure."

According to the documentation, the returned shared array is zeroed. https://docs.python.org/3.4/library/multiprocessing.html#module-multiprocessing.sharedctypes

In that case because the entire array is written at allocation, the process is expected to get killed if allocating more memory than available. Unless I am misunderstanding something, which is entirely possible.
msg215460 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-03 18:38
> Also, the FreeBSD man page for mmap() has the following warning:

That's mostly important for real file-backed mapping.
In our case, we don't want a file-backed mmap: we expect the mapping to fit
entirely in memory, so the writeback/read performance isn't that important
to us.

> Using truncate() to zero extend is not really portable: it is only
guaranteed on XSI-compliant POSIX systems.

Now that's annoying.
How about trying file.truncate() within a try block, and if an error is
raised fallback to the zero-filling?

Doing a lot of IO for an object which is supposed to be used for shared
memory is sad.

Or maybe it's time to add an API to access shared memory from Python (since
that's really what we're trying to achieve here).

> According to the documentation, the returned shared array is zeroed.
> In that case because the entire array is written at allocation, the
process is expected to get killed
> if allocating more memory than available. Unless I am misunderstanding
something, which is entirely
> possible.

Having the memory zero-filed doesn't require a write at all: when you do an
anonymous memory mapping for let's say 1Gb, the kernel doesn't
pre-emptively zero-fill it, it would be way to slow: usually it just sets
up the process page table to make this area a COW of a single zero page:
upon read, you'll read zeros, and upon write, it'll duplicate it as needed.

The only reason the code currently zero-fills the file is to avoid the
portability issues detailed by Richard.
msg215481 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-03 23:28
> Or maybe it's time to add an API to access shared memory from Python
> (since
> that's really what we're trying to achieve here).

That sounds like a good idea. Especially since we now have the memoryview type.
msg215494 - (view) Author: Médéric Boquien (mboquien) * Date: 2014-04-04 06:52
Thanks for the explanations Charles-François. I guess the new API would not be before 3.5 at least. Is there still a chance to integrate my patch (or any other) to improve the situation for the 3.4 series though?
msg215583 - (view) Author: Charles-François Natali (neologix) * (Python committer) Date: 2014-04-05 08:04
Indeed, I think it would make sense to consider this for 3.4, and even 2.7
if we opt for a simple fix.

As for the best way to fix it in the meantime, I'm fine with a buffered
zero-filling (the mere fact that noone ever complained until now probably
means that the performance isn't a show-stopper for users).
msg240704 - (view) Author: Roundup Robot (python-dev) Date: 2015-04-13 18:54
New changeset 0f944e424d67 by Antoine Pitrou in branch 'default':
Issue #21116: Avoid blowing memory when allocating a multiprocessing shared
https://hg.python.org/cpython/rev/0f944e424d67
msg240705 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-13 18:55
Ok, I've committed the patch. If desired, the generic API for shared memory can be tackled in a separate issue. Thank you Médéric!
msg240874 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-14 11:54
Instead of the loop you can use writelines():

    f.writelines([b'\0' * bs] * (size // bs))

It would be nice to add a comment that estimate why os.ftruncate() or seek+write can't be used here. At least a link to this issue with short estimation.
msg241027 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-04-14 20:58
Actually, recent POSIX states unconditionally that:

« If the file previously was smaller than this size, ftruncate() shall increase the size of the file. If the file size is increased, the extended area shall appear as if it were zero-filled. »

(from http://pubs.opengroup.org/onlinepubs/9699919799/functions/ftruncate.html)
History
Date User Action Args
2015-04-14 20:58:22pitrousetmessages: + msg241027
2015-04-14 11:54:59serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg240874
2015-04-13 18:55:22pitrousetstatus: open -> closed
resolution: fixed
messages: + msg240705

stage: patch review -> resolved
2015-04-13 18:54:28python-devsetnosy: + python-dev
messages: + msg240704
2015-04-13 18:35:32pitrousetversions: - Python 3.4
2014-04-05 08:04:42neologixsetmessages: + msg215583
2014-04-04 06:52:06mboquiensetmessages: + msg215494
2014-04-03 23:28:44pitrousetmessages: + msg215481
2014-04-03 18:38:00neologixsetmessages: + msg215460
2014-04-03 09:01:52mboquiensetmessages: + msg215433
2014-04-03 06:14:09neologixsetmessages: + msg215425
2014-04-02 23:20:53sbtsetmessages: + msg215408
2014-04-02 22:13:33mboquiensetmessages: + msg215407
2014-04-02 21:20:17neologixsetnosy: + neologix
messages: + msg215404
2014-04-01 07:03:42mboquiensetmessages: + msg215296
2014-03-31 21:18:48pitrousetnosy: + pitrou
messages: + msg215268
2014-03-31 20:24:20mboquiensetfiles: + shared_array.diff

messages: + msg215264
2014-03-31 19:58:15pitrousetnosy: + sbt
stage: patch review
type: resource usage

versions: + Python 3.5
2014-03-31 19:55:56mboquiensetfiles: - shared_array.diff
2014-03-31 19:55:41mboquiensetfiles: + shared_array.diff

messages: + msg215260
2014-03-31 19:37:47mboquiencreate