test_list uses unreasonable amounts of memory on 64-bit Linux #47357

hushp1pt · 2008-06-13T22:59:27Z

BPO	3107
Nosy	@loewis, @mhammond, @pitrou

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2008-08-22.09:14:00.552>
created_at = <Date 2008-06-13.22:59:26.758>
labels = ['performance']
title = 'test_list uses unreasonable amounts of memory on 64-bit Linux'
updated_at = <Date 2008-08-22.09:14:00.550>
user = 'https://bugs.python.org/hushp1pt'

bugs.python.org fields:

activity = <Date 2008-08-22.09:14:00.550>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2008-08-22.09:14:00.552>
closer = 'pitrou'
components = ['Demos and Tools']
creation = <Date 2008-06-13.22:59:26.758>
creator = 'hushp1pt'
dependencies = []
files = []
hgrepos = []
issue_num = 3107
keywords = []
message_count = 18.0
messages = ['68188', '68195', '68200', '68202', '68203', '68204', '68205', '68206', '68273', '68281', '68286', '68298', '71189', '71197', '71213', '71696', '71702', '71731']
nosy_count = 6.0
nosy_names = ['loewis', 'mhammond', 'Rhamphoryncus', 'pitrou', 'hushp1pt', 'jess']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue3107'
versions = ['Python 2.5']

hushp1pt · 2008-06-13T22:59:26Z

[tony@gossamer Python-2.5.1]$ ./configure
--prefix=/home/tony/root/usr/local/python-2.5.2 --enable-shared
--enable-static

[tony@gossamer bin]$ file python
python: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped
[tony@gossamer bin]$ uname -a
Linux gossamer.ambric.local 2.4.21-40.ELsmp #1 SMP Wed Mar 15 13:46:01
EST 2006 x86_64 x86_64 x86_64 GNU/Linux
[tony@gossamer bin]$ cat /etc/redhat-release
CentOS release 3.6 (Final)

loewis · 2008-06-14T00:14:12Z

How do you know that there is a memory leak?

hushp1pt · 2008-06-14T07:01:32Z

how do you know

Here is the story, sorry I skipped it before- I was at work then.

I was doing the basic build-from-source on RHEL (Centos) Linux, because
I don't have root and I need to install it in $HOME/something. I don't
try to change anything except the install location. See the ./configure
command given before.

I tried 2.5.2 because it was the latest + greatest. When I ran "make
test", it ran OK until it got to "test_list". Then it got stuck until
I killed it. Its very repeatable. In another window, "top" shows the
python process's memory growing and growing, until it owns basically all
available memory (8GB) after less than a minute or so.

Then I tried python 2.5.1, built exactly the same way, and make test
works OK- test_list completes in seconds.

loewis · 2008-06-14T07:31:10Z

Are you by any chance using gcc 4.3? That compiler is known to
miscompile Python, see bpo-3019 or bpo-2626.

Also, I'm puzzled that your prompt says Python-2.5.1 when configuring
with a prefix suggesting 2.5.2.

hushp1pt · 2008-06-14T08:15:53Z

are you using gcc 4.3
No, I don't think so.
[tony@gossamer tony]$ gcc --version
gcc (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-53)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And make is definitely using gcc, not something else.

prompt 2.5.1
Good eye. I grabbed the file-python info from python 2.5.1 after I
installed it (because 2.5.1 passed make test, I am planning to use
that). Its true, I should have got that from the python 2.5.2, which I
never did install- but unfortunately, it was deleted too soon. If it is
important I can rebuild 2.5.2 again.

loewis · 2008-06-14T08:19:27Z

Are you willing to debug this? Otherwise, I'll have to close this as
"works-for-me" (which it does, as it did for many other people).

hushp1pt · 2008-06-14T08:40:30Z

are you willing
Yes, so long as I don't need root, I can follow instructions OK.
By the way, the same thing (memory leak 2.5.2) occurred on Centos 4.6, a
different Linux box. Lets proceed on that Centos 4.6 box. Here are the
particulars:

Python-2.5.2]$ file python
python: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

[tony@hathi Python-2.5.2]$ gcc --version
gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[tony@hathi Python-2.5.2]$ cat /etc/redhat-release[tony@hathi CentOS
release 4.6 (Final)

[tony@hathi Python-2.5.2]$ uname -a
Linux hathi.ambric.local 2.6.9-67.0.15.ELsmp #1 SMP Thu May 8 10:50:20
EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

(after make test)
test_largefile
test_linuxaudiodev
test_linuxaudiodev skipped -- Use of the `audio' resource not enabled
test_list
(stalls here so I dumped it)
make: *** [test] Quit

(another window, just before I dumped it)
top - 01:36:31 up 4 days, 15:07, 5 users, load average: 2.70, 0.84, 0.32
Tasks: 87 total, 1 running, 86 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.8% us, 9.7% sy, 0.0% ni, 0.0% id, 86.0% wa, 0.0% hi, 0.5% si
Mem: 15639112k total, 15610836k used, 28276k free, 280k buffers
Swap: 24579440k total, 1533172k used, 23046268k free, 57676k cached

PID USER PR NI %CPU TIME+ %MEM VIRT RES SHR S COMMAND
17190 tony 24 0 17 1:05.49 90.9 32.2g 13g 7200 D python
69 root 16 0 6 0:12.97 0.0 0 0 0 D kswapd0

loewis · 2008-06-14T08:46:44Z

Run "python Lib/test/test_lists.py" separately, and find out which
specific test fails (to complete). Report that, and also try to separate
that test's code into its own file, and see whether you can reproduce it
there. If so, try to further strip it down as much as you can.

hushp1pt · 2008-06-16T17:25:37Z

make test not only fails "test_list", it also fails "test_tuple" and
"test_userlist". In all cases, the behavior looks the same -- memory
expands to > 90% and you kill it.

hushp1pt · 2008-06-16T18:19:39Z

in test_list.py, the following shows where it hit the memory leak:

[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_list.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/init.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/init.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/init.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc has bad mtime
import test.list_tests # from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
# wrote /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_addmul (main.ListTest) ... ok
test_append (main.ListTest) ... ok
Terminated

===========
in test_tuple.py, the following shows where it hit the memory leak:

[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_tuple.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/init.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/init.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/init.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_addmul (main.TupleTest) ... ok
Terminated

===========
in test_userlist.py, the following shows where it hit the memory leak:

[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_userlist.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/UserList.pyc matches
/home/tony/src/Python-2.5.2/Lib/UserList.py
import UserList # precompiled from
/home/tony/src/Python-2.5.2/Lib/UserList.pyc
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/init.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/init.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/init.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
import test.list_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_add_specials (main.UserListTest) ... ok
test_addmul (main.UserListTest) ... ok
test_append (main.UserListTest) ... ok
Terminated

==========

I commented out the following methods and reran the above tests.
(the following only shows the starting line of each method, but I
commented out the whole entire method)
Lib/test/list_tests.py: 226: # def test_append(self):
Lib/test/seq_tests.py: 248: # def test_addmul(self):

When I re-ran the three tests (test_list, test_tuple, test_userlist),
here is how they looked when I had to kill them:

(test_list)
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
Terminated

(test_tuple)
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
Terminated

(test_userlist)
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
import test.list_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_add_specials (main.UserListTest) ... ok
Terminated

That is all the info I have time to get right now. If I get time later
and no other instructions, I would try rebuilding without enable-dynamic
or enable-static in the ./configure options

hushp1pt · 2008-06-16T18:51:25Z

tried again with
/configure --prefix=/home/tony/root/usr/local/python-2.5.2 --with-tcl
--disable-shared

No change

But I noticed this when it recompiled. Maybe it is related.

gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes -I. -IInclude -I./Include -DPy_BUILD_CORE -o
Objects/obmalloc.o Objects/obmalloc.c
Objects/obmalloc.c: In function `new_arena':
Objects/obmalloc.c:529: warning: comparison is always false due to
limited range of data type

(code fragment)

    /* Double the number of arena objects on each allocation.
     * Note that it's possible for `numarenas` to overflow.
     */
    numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS;
    if (numarenas <= maxarenas)
        return NULL;    /* overflow */
    if (numarenas > PY_SIZE_MAX / sizeof(*arenas)) /* line 529 here */
        return NULL;    /* overflow */
    nbytes = numarenas * sizeof(*arenas);
    arenaobj = (struct arena_object *)realloc(arenas, nbytes);
    if (arenaobj == NULL)
        return NULL;
    arenas = arenaobj;

hushp1pt · 2008-06-17T00:01:45Z

Objects/obmalloc.c:529: warning:
comparison is always false due to limited range of data type

This compile complaint was definitely introduced in 2.5.2 by source
changes from 2.5.1. So, there's a minor problem that could be fixed,
anyway.

However, replacing the 2.5.2 version of obmalloc.c with the 2.5.1
version and rebuilding (with incremental make this time) did NOT help
the memory leak in test_list - I still get it.

jess · 2008-08-15T22:36:53Z

This appears to be the same issue as in:
3055 2 months ago test_list on 64-bit platforms

The failing test appears to be test_bigrepeat:

    def test_bigrepeat(self):
        x = self.type2test([0])
        x *= 2**16
        self.assertRaises(MemoryError, x.__mul__, 2**16)
        if hasattr(x, '__imul__'):
            self.assertRaises(MemoryError, x.__imul__, 2**16)

I am experiencing the same symptoms with 64-bit builds on Solaris10 on
an 8gb sparc machine with lots of virtual memory. The test is
attempting to allocate a list with 4gb elements. This requires more
space than there is physical memory so the machine starts swapping like
crazy but does not throw an error in 64 bit mode. In 32 bit, it would
presumabley throw the error because the test has requested more than 4gb
of memory.

In other words Python appears to be working as it should. In my case I
am not sure that Solaris10 is scanning for virtual memory properly but
that is an entirely different problem.

loewis · 2008-08-16T06:20:09Z

Ok, so it seems there is no memory leak. Changing the title accordingly.

pitrou · 2008-08-16T13:29:53Z

Apparently this has been fixed as part of r65335 ("Security patches from
Apple: prevent int overflow when allocating memory"), although it uses
sys.maxint where sys.maxsize may be more appropriate. Can you check?

hushp1pt · 2008-08-21T23:20:41Z

It worked- I took a patch of r65334, as
svn diff -c 65334
"http://svn.python.org/projects/python/branches/release25-maint"

and applied that patch ONLY to a clean release 2.5.2 source, ignoring
the patch failure in Misc/NEWS. Built it all over again the same as
before (that is,
CentOS release 4.4 (Final)
Linux manfred 2.6.9-42.0.8.ELsmp #1 SMP Tue Jan 30 12:18:01 EST 2007
x86_64 x86_64 x86_64 GNU/Linux
./configure --prefix=/home/tools/sqa/amd64_rhel4/Python-2.5.2
--enable-shared --build=x86_64-redhat-linux --enable-static )

Now, make test runs all the way through with no difficulty.

Thanks, everyone. I consider this closed. Maintainer, please dispose
this bug as you think appropriate.

mhammond · 2008-08-21T23:41:58Z

It looks like I made a dupe at http://bugs.python.org/issue3625, where I
reported the same thing on 64bit windows (and 2 other cases that I'd be
interested to know if cause problems for you too...)

pitrou · 2008-08-22T09:14:00Z

Thanks, Tony!

hushp1pt mannequin added the performance Performance or resource usage label Jun 13, 2008

loewis mannequin changed the title ~~memory leak in make test (in "test list"), 2.5.2 not 2.5.1, Linux 64bit~~ test_list uses unreasonable amounts of memory on 64-bit Linux Aug 16, 2008

pitrou closed this as completed Aug 22, 2008

ezio-melotti transferred this issue from another repository Apr 10, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_list uses unreasonable amounts of memory on 64-bit Linux #47357

test_list uses unreasonable amounts of memory on 64-bit Linux #47357

hushp1pt mannequin commented Jun 13, 2008

hushp1pt mannequin commented Jun 13, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 17, 2008

jess mannequin commented Aug 15, 2008

loewis mannequin commented Aug 16, 2008

pitrou commented Aug 16, 2008

hushp1pt mannequin commented Aug 21, 2008

mhammond commented Aug 21, 2008

pitrou commented Aug 22, 2008

test_list uses unreasonable amounts of memory on 64-bit Linux #47357

test_list uses unreasonable amounts of memory on 64-bit Linux #47357

Comments

hushp1pt mannequin commented Jun 13, 2008

hushp1pt mannequin commented Jun 13, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 14, 2008

loewis mannequin commented Jun 14, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 16, 2008

hushp1pt mannequin commented Jun 17, 2008

jess mannequin commented Aug 15, 2008

loewis mannequin commented Aug 16, 2008

pitrou commented Aug 16, 2008

hushp1pt mannequin commented Aug 21, 2008

mhammond commented Aug 21, 2008

pitrou commented Aug 22, 2008