Author njs
Recipients josh.r, jtaylor, neologix, njs, pitrou, skrah, vstinner
Date 2014-04-27.18:27:30
SpamBayes Score -1.0
Marked as misclassified Yes
Message-id <1398623251.49.0.459241305179.issue21233@psf.upfronthosting.co.za>
In-reply-to
Content
Right, python3 -c 'b"x" * (2 ** 48)' does give an instant MemoryError for me. So I was wrong about it being the VM limit indeed.

The documentation on this is terrible! But, if I'm reading this right:
   http://lxr.free-electrons.com/source/mm/util.c#L434
the actual rules are:

overcommit mode 1: allocating a VM range always succeeds.
overcommit mode 2: (Slightly simplified) You can allocate total VM ranges up to (swap + RAM * overcommit_ratio), and overcommit_ratio is 50% by default. So that's a bit odd, but whatever. This is still entirely a limit on VM size.
overcommit mode 0 ("guess", the default): when allocating a VM range, the kernel imagines what would happen if you immediately used all those pages. If that would put you OOM, then we fall back to mode 2 rules. If that would *not* put you OOM, then the allocation unconditionally succeeds.

So yeah, touching pages can affect whether a later malloc returns ENOMEM.

I'm not sure any of this actually matters in the Python case though :-). There's still no reason to go touching pages pre-emptively just in case we might write to them later -- all that does is increase the interpreter's memory footprint, which can't help anything. If people are worried about overcommit, then they should turn off overcommit, not try and disable it on a piece-by-piece basis by trying to get individual programs to memory before they need it.
History
Date User Action Args
2014-04-27 18:27:31njssetrecipients: + njs, pitrou, vstinner, skrah, neologix, jtaylor, josh.r
2014-04-27 18:27:31njssetmessageid: <1398623251.49.0.459241305179.issue21233@psf.upfronthosting.co.za>
2014-04-27 18:27:31njslinkissue21233 messages
2014-04-27 18:27:30njscreate