classification
Title: test_bigmem.test_from_2G_generator uses more memory than expected
Type: resource usage Stage: resolved
Components: Tests Versions: Python 3.8, Python 3.7
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: twouters Nosy List: llllllllll, miss-islington, nadeem.vawda, pitrou, serhiy.storchaka, steve.dower, twouters
Priority: normal Keywords: patch

Created on 2009-03-08 01:25 by pitrou, last changed 2019-08-02 22:25 by steve.dower. This issue is now closed.

Files
File name Uploaded Description Edit
bigmem.patch llllllllll, 2015-04-13 19:41 review
bigmem-update.patch llllllllll, 2015-04-13 20:15 review
bigmem_usage_440G.png steve.dower, 2015-04-14 17:02
Pull Requests
URL Status Linked Edit
PR 11123 merged serhiy.storchaka, 2018-12-11 22:30
PR 11259 merged miss-islington, 2018-12-20 07:35
Messages (22)
msg83295 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-08 01:25
It seems test_from_2G_generator doesn't respect its announced memory
consumption. I launched test_bigmem with "-M 12G" on a 16GB machine, and
the interpreter process got killed by the OOM killer (Out Of Memory)
while running this test case.
msg109768 - (view) Author: Mark Lawrence (BreamoreBoy) * Date: 2010-07-09 16:40
Is this problem repeatable?  I haven't got a big enough box on which to try this out.
msg132074 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2011-03-25 09:31
I'm not surprised this test takes up more memory... It claims to take up 9 times the test size in memory, with a minimum useful test size of 1G. With a limit of 12G that would make the test size about 1.33G. However, it then proceeds to build a tuple of 159 million distinct integers and stuffs them into a tuple. Each integer is at least 24 bytes long, plus the 8 of the pointer in the tuple, making the total memory use over 32 times the test size. I'll fix the test.
msg240713 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-13 19:41
I am now computing the size based on the system's int and pointer size.
msg240728 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-13 20:15
I have updated the test to make another test that does not have a __len__.
msg240842 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-14 05:11
Steve, we talked about the possibility of having you run some of these tests on a machine with a large amount of ram; I am adding you to the nosy list.
msg240848 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-04-14 06:18
How long time the test run (if there is enough memory)? May be we should decrease memory requirements by using shared values?
msg240870 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-14 11:03
I am not sure yet; I don't have access to a machine that can run the test.
msg240884 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 13:38
I'll soon have Windows and Ubuntu machines with 448GB RAM each - anything special I should test here?
msg240887 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-14 13:52
The tests need to be run with higher memory caps, this can be set with the -M flag, like "python -m test -M 64G test_bigmem". I am under the impression that these tests don't get run all the time so we might want to run that suite.
msg240913 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 15:37
It's running fine, but not really exercising the limit. I ran with -M 440G and we haven't gone past 38GB usage yet. I'll post a timeline plot when it finishes (on Windows - not sure how to collect the data but I'm also running on Ubuntu), but it looks like we may need to tweak the tests to go closer to the limit.
msg240922 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 15:59
I'm up to test_from_2G_generator. It says the peak will be 36GB. Memory grew in chunks up to about 40GB and has been stuck there for 10 minutes.
msg240936 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-14 16:31
Has the test passed or is it still hung? Also, I guess this means that the tuple is over-allocating, maybe I need to revise the memory assumptions to account for the growth algorithm.
msg240944 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 16:51
It's passed. There was no memory usage increase from that point, but after about 20 minutes it completed successfully.

All the tests passed on Ubuntu, but again, they didn't really fill up RAM - 50GB looks like the high point.
msg240953 - (view) Author: Steve Dower (steve.dower) * (Python committer) Date: 2015-04-14 17:02
Just for fun, I attached the memory usage graph from the Windows run. You can see the long pauses in the middle for the two changed tests, and the graph only goes up to 57GB despite the limit being set at 440GB.
msg240960 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-14 17:16
I think that the idea is not to get as close to the limit as possible but just to hard cap the memory usage of the test suite so that it doesn't get oom-killed. twouters, does this sound correct? Also, I think that this means that the new decorator is reporting a proper size hint.
msg240968 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2015-04-14 18:08
Yes. The test doesn't run with more than the requested "units", but if you
run it without -M it does run with a very small number, in order to make
sure the tests don't bitrot.

The mechanism could do with an overhaul though; I wrote it in 2006 or so,
before we had test skipping and when memory limits were a lot lower (and we
had fewer, less powerful friends :) I also wonder if we should change it to
run each test in a separate process so we can more easily check the actual
memory usage.
On Apr 14, 2015 1:16 PM, "Joe Jevnik" <report@bugs.python.org> wrote:

>
> Joe Jevnik added the comment:
>
> I think that the idea is not to get as close to the limit as possible but
> just to hard cap the memory usage of the test suite so that it doesn't get
> oom-killed. twouters, does this sound correct? Also, I think that this
> means that the new decorator is reporting a proper size hint.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue5438>
> _______________________________________
>
msg241147 - (view) Author: Joe Jevnik (llllllllll) * Date: 2015-04-15 19:20
I can look into rewriting the framework to use multiprocessing; however, should we split this into a separate issue or block this one until that work is done.
msg241939 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2015-04-24 13:15
Rewriting the tests shouldn't block this specific issue, no. Also, don't use multiprocessing for it. I would just use subprocess to start a separate process (which might after all be OOM-killed,) check the exitcode, and record its stderr in case of failure. Given that Steve ran the bigmem tests with a high limit and they all passed means that it's not pressing, though -- while it would be *good* to make these tests easier, and the skips more obvious, it's a good sign that nothing broke :)
msg331660 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-11 22:26
tuple(range(size)) should consume approximately 90 GiB of memory and run for hours. I think this happens because it creates enormous number of integer objects, and this can lead to requesting additional memory for memory pools.

If details of memory management for small objects is not the purpose of this test, I think we can replace it with tuple(iter([42]*size)). It executes the same code in tuple creation: allocates a tuple of known size (both range and list iterators have __length_hint__) and fills it with values produced by the iterator. This allows significantly reduce memory requirements (to 16 GiB) and execution time (to 20 seconds on my computer).

The proposed PR implements this rewriting. It also fixes memory requirements for other tests, optimizes repr tests for tuples and lists (repr(False) is faster and longer than repr(0)), and expresses memory requirements in terms of character and pointer sizes.
msg332191 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2018-12-20 07:34
New changeset b13a20f50789e153c18ed8efb4fbc5eecc50f2cd by Serhiy Storchaka in branch 'master':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
https://github.com/python/cpython/commit/b13a20f50789e153c18ed8efb4fbc5eecc50f2cd
msg332194 - (view) Author: miss-islington (miss-islington) Date: 2018-12-20 07:52
New changeset a26201cd8ef17dc81431f768846291c9f4337550 by Miss Islington (bot) in branch '3.7':
bpo-5438: Update memory requirements and optimize test_bigmem.py. (GH-11123)
https://github.com/python/cpython/commit/a26201cd8ef17dc81431f768846291c9f4337550
History
Date User Action Args
2019-08-02 22:25:50steve.dowersetstatus: open -> closed
resolution: fixed
stage: patch review -> resolved
2018-12-20 07:52:59miss-islingtonsetnosy: + miss-islington
messages: + msg332194
2018-12-20 07:35:01miss-islingtonsetpull_requests: + pull_request10489
2018-12-20 07:34:53serhiy.storchakasetmessages: + msg332191
2018-12-11 22:30:13serhiy.storchakasetstage: patch review
pull_requests: + pull_request10355
2018-12-11 22:26:51serhiy.storchakasetmessages: + msg331660
versions: + Python 3.7, Python 3.8, - Python 3.3
2015-04-24 13:15:25twouterssetmessages: + msg241939
2015-04-15 19:20:05llllllllllsetmessages: + msg241147
2015-04-14 18:08:13twouterssetmessages: + msg240968
2015-04-14 17:16:18llllllllllsetmessages: + msg240960
2015-04-14 17:02:06steve.dowersetfiles: + bigmem_usage_440G.png

messages: + msg240953
2015-04-14 16:51:30steve.dowersetmessages: + msg240944
2015-04-14 16:31:33llllllllllsetmessages: + msg240936
2015-04-14 15:59:57steve.dowersetmessages: + msg240922
2015-04-14 15:37:18steve.dowersetmessages: + msg240913
2015-04-14 13:52:56llllllllllsetmessages: + msg240887
2015-04-14 13:38:06steve.dowersetmessages: + msg240884
2015-04-14 11:03:46llllllllllsetmessages: + msg240870
2015-04-14 06:18:16serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg240848
2015-04-14 05:11:00llllllllllsetnosy: + steve.dower
messages: + msg240842
2015-04-13 20:15:49llllllllllsetfiles: + bigmem-update.patch

messages: + msg240728
2015-04-13 19:41:16llllllllllsetfiles: + bigmem.patch

nosy: + llllllllll
messages: + msg240713

keywords: + patch
2014-02-03 19:14:05BreamoreBoysetnosy: - BreamoreBoy
2012-02-24 15:03:51nadeem.vawdasetnosy: + nadeem.vawda
2011-03-25 09:31:41twouterssetversions: + Python 3.3, - Python 3.1, Python 2.7, Python 3.2
nosy: + twouters

messages: + msg132074

assignee: twouters
2010-07-09 16:40:04BreamoreBoysetnosy: + BreamoreBoy

messages: + msg109768
versions: + Python 2.7, Python 3.2
2009-03-08 01:25:13pitroucreate