>> Hm...
>> What's /proc/sys/vm/overcommit_memory ?
>> If it's set to 0, then the kernel will always overcommit.
>
> Ah, indeed.

See above, I mistyped: 0 is the default (which is already quite
optimistic), 1 is always.

>> If you set it to 2, normally you'd definitely get ENOMEM
>
> You're right, but with weird results:
>
> $ gcc -o /tmp/test test.c; /tmp/test
> malloc() returned NULL after 600MB
> $ gcc -DDO_MEMSET -o /tmp/test test.c; /tmp/test
> malloc() returned NULL after 600MB
>
> (I'm supposed to have gigabytes free?!)

The formula is RAM * vm.overcommit_ratio /100 + swap

So if you don't have swap, or a low overcommit_ratio, it could explain
why it returns so early.
Or maybe you have some processes with a lot of mapped-yet-unused
memory (chromium is one of those for example).

Anyway, it's really a mess!