[tony@gossamer Python-2.5.1]$ ./configure
--prefix=/home/tony/root/usr/local/python-2.5.2 --enable-shared
--enable-static

[tony@gossamer bin]$ file python
python: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped
[tony@gossamer bin]$  uname -a
Linux gossamer.ambric.local 2.4.21-40.ELsmp #1 SMP Wed Mar 15 13:46:01
EST 2006 x86_64 x86_64 x86_64 GNU/Linux
[tony@gossamer bin]$ cat /etc/redhat-release
CentOS release 3.6 (Final)

How do you know that there is a memory leak?

> how do you know

Here is the story, sorry I skipped it before- I was at work then. 

I was doing the basic build-from-source on RHEL (Centos) Linux, because
I don't have root and I need to install it in $HOME/something.  I don't
try to change anything except the install location.  See the ./configure
command given before. 

I tried 2.5.2 because it was the latest + greatest.  When I ran "make
test",  it ran OK until it got to "test_list".  Then it got stuck until
I killed it.  Its very repeatable.  In another window, "top" shows the
python process's memory growing and growing, until it owns basically all
available memory (8GB) after less than a minute or so.  

Then I tried python 2.5.1, built exactly the same way, and make test
works OK- test_list completes in seconds.

Are you by any chance using gcc 4.3? That compiler is known to
miscompile Python, see issue3019 or issue2626.

Also, I'm puzzled that your prompt says Python-2.5.1 when configuring
with a prefix suggesting 2.5.2.

> are you using gcc 4.3
No, I don't think so. 
[tony@gossamer tony]$ gcc --version
gcc (GCC) 3.2.3 20030502 (Red Hat Linux 3.2.3-53)
Copyright (C) 2002 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And make is definitely using gcc, not something else.


> prompt 2.5.1
Good eye. I grabbed the file-python info from python 2.5.1 after I
installed it (because 2.5.1 passed make test, I am planning to use
that).  Its true, I should have got that from the python 2.5.2, which I
never did install- but unfortunately, it was deleted too soon.  If it is
important I can rebuild 2.5.2 again.

Are you willing to debug this? Otherwise, I'll have to close this as
"works-for-me" (which it does, as it did for many other people).

> are you willing
Yes, so long as I don't need root, I can follow instructions OK.
By the way, the same thing (memory leak 2.5.2) occurred on Centos 4.6, a
different Linux box.  Lets proceed on that Centos 4.6 box.  Here are the
particulars:

Python-2.5.2]$ file python
python: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV), for
GNU/Linux 2.4.0, dynamically linked (uses shared libs), not stripped

[tony@hathi Python-2.5.2]$ gcc --version
gcc (GCC) 3.4.6 20060404 (Red Hat 3.4.6-9)
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[tony@hathi Python-2.5.2]$ cat /etc/redhat-release[tony@hathi CentOS
release 4.6 (Final)

[tony@hathi Python-2.5.2]$ uname -a
Linux hathi.ambric.local 2.6.9-67.0.15.ELsmp #1 SMP Thu May 8 10:50:20
EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

(after make test)
test_largefile
test_linuxaudiodev
test_linuxaudiodev skipped -- Use of the `audio' resource not enabled
test_list
(stalls here so I dumped it)
make: *** [test] Quit

(another window, just before I dumped it)
top - 01:36:31 up 4 days, 15:07,  5 users,  load average: 2.70, 0.84, 0.32
Tasks:  87 total,   1 running,  86 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.8% us,  9.7% sy,  0.0% ni,  0.0% id, 86.0% wa,  0.0% hi,  0.5% si
Mem:  15639112k total, 15610836k used,    28276k free,      280k buffers
Swap: 24579440k total,  1533172k used, 23046268k free,    57676k cached

  PID USER      PR  NI %CPU    TIME+  %MEM  VIRT  RES  SHR S COMMAND
17190 tony      24   0   17   1:05.49 90.9 32.2g  13g 7200 D python
   69 root      16   0    6   0:12.97  0.0     0    0    0 D kswapd0

Run "python Lib/test/test_lists.py" separately, and find out which
specific test fails (to complete). Report that, and also try to separate
that test's code into its own file, and see whether you can reproduce it
there. If so, try to further strip it down as much as you can.

make test not only fails "test_list", it also fails "test_tuple" and
"test_userlist".  In all cases, the behavior looks the same -- memory
expands to > 90% and you kill it.

in test_list.py, the following shows where it hit the memory leak:

[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_list.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/__init__.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/__init__.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/__init__.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc has bad mtime
import test.list_tests # from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
# wrote /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_addmul (__main__.ListTest) ... ok
test_append (__main__.ListTest) ... ok
Terminated

===========
in test_tuple.py, the following shows where it hit the memory leak:


[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_tuple.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/__init__.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/__init__.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/__init__.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_addmul (__main__.TupleTest) ... ok
Terminated


===========
in test_userlist.py, the following shows where it hit the memory leak:

[tony@hathi Python-2.5.2]$
LD_LIBRARY_PATH=/home/tony/src/Python-2.5.2/Lib/:$LD_LIBRARY_PATH
./python -v Lib/test/test_userlist.py
# installing zipimport hook
import zipimport # builtin
<<...>>
Python 2.5.2 (r252:60911, Jun 14 2008, 01:31:25)
[GCC 3.4.6 20060404 (Red Hat 3.4.6-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
# /home/tony/src/Python-2.5.2/Lib/UserList.pyc matches
/home/tony/src/Python-2.5.2/Lib/UserList.py
import UserList # precompiled from
/home/tony/src/Python-2.5.2/Lib/UserList.pyc
# /home/tony/src/Python-2.5.2/Lib/unittest.pyc matches
/home/tony/src/Python-2.5.2/Lib/unittest.py
import unittest # precompiled from
/home/tony/src/Python-2.5.2/Lib/unittest.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so", 2);
import time # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/time.so
# /home/tony/src/Python-2.5.2/Lib/traceback.pyc matches
/home/tony/src/Python-2.5.2/Lib/traceback.py
import traceback # precompiled from
/home/tony/src/Python-2.5.2/Lib/traceback.pyc
import test # directory /home/tony/src/Python-2.5.2/Lib/test
# /home/tony/src/Python-2.5.2/Lib/test/__init__.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/__init__.py
import test # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/__init__.pyc
# /home/tony/src/Python-2.5.2/Lib/test/test_support.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/test_support.py
import test.test_support # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/test_support.pyc
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
import test.list_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_add_specials (__main__.UserListTest) ... ok
test_addmul (__main__.UserListTest) ... ok
test_append (__main__.UserListTest) ... ok
Terminated

==========

I commented out the following methods and reran the above tests. 
(the following only shows the starting line of each method, but I 
commented out the whole entire method)
Lib/test/list_tests.py: 226:    #   def test_append(self):
Lib/test/seq_tests.py: 248:     #   def test_addmul(self):

When I re-ran the three tests (test_list, test_tuple, test_userlist),
here is how they looked when I had to kill them:

(test_list)
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
Terminated


(test_tuple)
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
Terminated

(test_userlist)
# /home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/list_tests.py
import test.list_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/list_tests.pyc
# /home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc matches
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.py
import test.seq_tests # precompiled from
/home/tony/src/Python-2.5.2/Lib/test/seq_tests.pyc
dlopen("/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so",
2);
import itertools # dynamically loaded from
/home/tony/src/Python-2.5.2/build/lib.linux-x86_64-2.5/itertools.so
test_add_specials (__main__.UserListTest) ... ok
Terminated

That is all the info I have time to get right now.  If I get time later
and no other instructions, I would try rebuilding without enable-dynamic
or enable-static in the ./configure options

tried again with
/configure --prefix=/home/tony/root/usr/local/python-2.5.2 --with-tcl
--disable-shared

No change

But I noticed this when it recompiled.  Maybe it is related.

gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall
-Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -o
Objects/obmalloc.o Objects/obmalloc.c
Objects/obmalloc.c: In function `new_arena':
Objects/obmalloc.c:529: warning: comparison is always false due to
limited range of data type

(code fragment)

        /* Double the number of arena objects on each allocation.
         * Note that it's possible for `numarenas` to overflow.
         */
        numarenas = maxarenas ? maxarenas << 1 : INITIAL_ARENA_OBJECTS;
        if (numarenas <= maxarenas)
            return NULL;    /* overflow */
        if (numarenas > PY_SIZE_MAX / sizeof(*arenas)) /* line 529 here */
            return NULL;    /* overflow */
        nbytes = numarenas * sizeof(*arenas);
        arenaobj = (struct arena_object *)realloc(arenas, nbytes);
        if (arenaobj == NULL)
            return NULL;
        arenas = arenaobj;

> Objects/obmalloc.c:529: warning: 
> comparison is always false due to limited range of data type

This compile complaint was definitely introduced in 2.5.2 by source
changes from 2.5.1.  So, there's a minor problem that could be fixed,
anyway. 

However, replacing the 2.5.2 version of obmalloc.c with the 2.5.1
version and rebuilding (with incremental make this time) did NOT help
the memory leak in test_list - I still get it.

This appears to be the same issue as in:
3055  	2 months ago  	 test_list on 64-bit platforms

The failing test appears to be test_bigrepeat:

    def test_bigrepeat(self):
        x = self.type2test([0])
        x *= 2**16
        self.assertRaises(MemoryError, x.__mul__, 2**16)
        if hasattr(x, '__imul__'):
            self.assertRaises(MemoryError, x.__imul__, 2**16)

I am experiencing the same symptoms with 64-bit builds on Solaris10 on
an 8gb sparc machine with lots of virtual memory.  The test is
attempting to allocate a list with 4gb elements.  This requires more
space than there is physical memory so the machine starts swapping like
crazy but does not throw an error in 64 bit mode.  In 32 bit, it would
presumabley throw the error because the test has requested more than 4gb
of memory.

In other words Python appears to be working as it should.  In my case I
am not sure that Solaris10 is scanning for virtual memory properly but
that is an entirely different problem.

Ok, so it seems there is no memory leak. Changing the title accordingly.

Apparently this has been fixed as part of r65335 ("Security patches from
Apple:  prevent int overflow when allocating memory"), although it uses
sys.maxint where sys.maxsize may be more appropriate. Can you check?

It worked- I took a patch of r65334, as 
 svn diff -c 65334
"http://svn.python.org/projects/python/branches/release25-maint"

and applied that patch ONLY to a clean release 2.5.2 source, ignoring
the patch failure in Misc/NEWS.  Built it all over again the same as
before (that is, 
 CentOS release 4.4 (Final)
 Linux manfred 2.6.9-42.0.8.ELsmp #1 SMP Tue Jan 30 12:18:01 EST 2007
x86_64 x86_64 x86_64 GNU/Linux 
 ./configure --prefix=/home/tools/sqa/amd64_rhel4/Python-2.5.2
--enable-shared  --build=x86_64-redhat-linux --enable-static )

Now, make test runs all the way through with no difficulty. 

Thanks, everyone.  I consider this closed.  Maintainer, please dispose
this bug as you think appropriate.

It looks like I made a dupe at http://bugs.python.org/issue3625, where I
reported the same thing on 64bit windows (and 2 other cases that I'd be
interested to know if cause problems for you too...)

Thanks, Tony!