It seems test_from_2G_generator doesn't respect its announced memory
consumption. I launched test_bigmem with "-M 12G" on a 16GB machine, and
the interpreter process got killed by the OOM killer (Out Of Memory)
while running this test case.

Is this problem repeatable? I haven't got a big enough box on which to try this out.

I'm not surprised this test takes up more memory... It claims to take up 9 times the test size in memory, with a minimum useful test size of 1G. With a limit of 12G that would make the test size about 1.33G. However, it then proceeds to build a tuple of 159 million distinct integers and stuffs them into a tuple. Each integer is at least 24 bytes long, plus the 8 of the pointer in the tuple, making the total memory use over 32 times the test size. I'll fix the test.

I am now computing the size based on the system's int and pointer size.

I have updated the test to make another test that does not have a __len__.

Steve, we talked about the possibility of having you run some of these tests on a machine with a large amount of ram; I am adding you to the nosy list.

How long time the test run (if there is enough memory)? May be we should decrease memory requirements by using shared values?

I am not sure yet; I don't have access to a machine that can run the test.

I'll soon have Windows and Ubuntu machines with 448GB RAM each - anything special I should test here?

The tests need to be run with higher memory caps, this can be set with the -M flag, like "python -m test -M 64G test_bigmem". I am under the impression that these tests don't get run all the time so we might want to run that suite.

It's running fine, but not really exercising the limit. I ran with -M 440G and we haven't gone past 38GB usage yet. I'll post a timeline plot when it finishes (on Windows - not sure how to collect the data but I'm also running on Ubuntu), but it looks like we may need to tweak the tests to go closer to the limit.

I'm up to test_from_2G_generator. It says the peak will be 36GB. Memory grew in chunks up to about 40GB and has been stuck there for 10 minutes.

Has the test passed or is it still hung? Also, I guess this means that the tuple is over-allocating, maybe I need to revise the memory assumptions to account for the growth algorithm.

It's passed. There was no memory usage increase from that point, but after about 20 minutes it completed successfully.

All the tests passed on Ubuntu, but again, they didn't really fill up RAM - 50GB looks like the high point.

Just for fun, I attached the memory usage graph from the Windows run. You can see the long pauses in the middle for the two changed tests, and the graph only goes up to 57GB despite the limit being set at 440GB.

I think that the idea is not to get as close to the limit as possible but just to hard cap the memory usage of the test suite so that it doesn't get oom-killed. twouters, does this sound correct? Also, I think that this means that the new decorator is reporting a proper size hint.

Yes. The test doesn't run with more than the requested "units", but if you
run it without -M it does run with a very small number, in order to make
sure the tests don't bitrot.

The mechanism could do with an overhaul though; I wrote it in 2006 or so,
before we had test skipping and when memory limits were a lot lower (and we
had fewer, less powerful friends :) I also wonder if we should change it to
run each test in a separate process so we can more easily check the actual
memory usage.
On Apr 14, 2015 1:16 PM, "Joe Jevnik" <report@bugs.python.org> wrote:

Joe Jevnik added the comment:

I think that the idea is not to get as close to the limit as possible but
just to hard cap the memory usage of the test suite so that it doesn't get
oom-killed. twouters, does this sound correct? Also, I think that this
means that the new decorator is reporting a proper size hint.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue5438\>

---

I can look into rewriting the framework to use multiprocessing; however, should we split this into a separate issue or block this one until that work is done.

Rewriting the tests shouldn't block this specific issue, no. Also, don't use multiprocessing for it. I would just use subprocess to start a separate process (which might after all be OOM-killed,) check the exitcode, and record its stderr in case of failure. Given that Steve ran the bigmem tests with a high limit and they all passed means that it's not pressing, though -- while it would be *good* to make these tests easier, and the skips more obvious, it's a good sign that nothing broke :)

tuple(range(size)) should consume approximately 90 GiB of memory and run for hours. I think this happens because it creates enormous number of integer objects, and this can lead to requesting additional memory for memory pools.

If details of memory management for small objects is not the purpose of this test, I think we can replace it with tuple(iter([42]*size)). It executes the same code in tuple creation: allocates a tuple of known size (both range and list iterators have __length_hint__) and fills it with values produced by the iterator. This allows significantly reduce memory requirements (to 16 GiB) and execution time (to 20 seconds on my computer).

The proposed PR implements this rewriting. It also fixes memory requirements for other tests, optimizes repr tests for tuples and lists (repr(False) is faster and longer than repr(0)), and expresses memory requirements in terms of character and pointer sizes.

New changeset b13a20f by Serhiy Storchaka in branch 'master':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
b13a20f

New changeset a26201c by Miss Islington (bot) in branch '3.7':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
a26201c