Hi Stefan,
In your message https://bugs.python.org/issue41540#msg375462 , you said:
 "However, instead of freezing the machine, the process gets a proper SIGKILL almost instantly."
That's probably due to a very small size of the Paging Space of the AIX machine you used for testing. With very small PS, the OS quickly reaches the step where PS and memory are full and it tries to kill possible culprits (but often killing innocent processes, like my bash shell). However, with a large PS (size of the Memory, or half), it takes some time for the OS to consume the PS, and, during this time (many seconds if not minutes), the OS looks like frozen and it takes many seconds or minutes for a "kill -9 PID" to take effect.

About -bmaxdata, I always used it for extending default memory of a 32bit process, but I never used it for reducing the possible memory of a 64bit process since some users may want to use python with hundreds of GigaBytes of memory. And the python executable used for tests is the same one that is delivered to users.

About PSALLOC=early , I confirm that it perfectly fixes the issue. So, we'll use it when testing Python.
Our customers should use it or use ulimit -d .
But using -bmaxdata for building python process in 64bit would reduce the possibilities of the python process.
In the future, we'll probably improve the compatibility with Linux so that this (rare) case no more appear.

BTW, on AIX, we have only 12 test cases failing out of about 32,471 test cases run in 64bit, with probably only 5 remaining serious failures. Both with GCC and XLC. Not bad. Less in 32bit. Now studying these few remaining issues and the still skipped tests.