Author kaifeng
Recipients flox, kaifeng, neologix, pitrou
Date 2011-04-19.02:41:36
SpamBayes Score 1.096e-07
Marked as misclassified No
Message-id <1303180899.72.0.186167652301.issue11849@psf.upfronthosting.co.za>
In-reply-to
Content
I applied your patch to Python 3.2, also I added a function call to 'malloc_trim' via ctypes, as you can see in issue11849_test2.py.

In fact I have a daemon written in Python 2.5, parsing an XML of size 10+ MB every 5 minutes, after 16+ hours running, the program finally exhausted 4 GB memory and died.  I simplified the logic of the daemon and found ElementTree eats too much memory.  There comes the attached test script.

BTW, after utilize lxml instead of ElementTree, such phenomenon of increasing memory usage disappeared.


$ ltrace -e malloc_trim python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
--- SIGCHLD (Child exited) ---
*** Python 3.2.0 final
--- SIGCHLD (Child exited) ---
---   PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
--- SIGCHLD (Child exited) ---
  0 13708 pts/1    S+     0:00      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:00      1  1316 11055  6440  0.6 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  1 13708 pts/1    S+     0:00      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:03      1  1316 53155 47332  4.5 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  2 13708 pts/1    S+     0:00      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:06      1  1316 91055 85204  8.2 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  3 13708 pts/1    S+     0:01      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:10      1  1316 128947 124212 11.9 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  4 13708 pts/1    S+     0:01      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:13      1  1316 166807 162280 15.6 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  5 13708 pts/1    S+     0:01      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:16      1  1316 204483 198808 19.2 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  6 13708 pts/1    S+     0:02      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:20      1  1316 242379 236672 22.8 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  7 13708 pts/1    S+     0:02      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:23      1  1316 284383 277508 26.8 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  8 13708 pts/1    S+     0:03      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:27      1  1316 318191 312436 30.1 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
  9 13708 pts/1    S+     0:03      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:29      1  1316 360199 353272 34.1 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
END 13708 pts/1    S+     0:03      1    65  1742   636  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:34      1  1316 393975 388164 37.4 python3.2 Issue11849_test2.py
malloc_trim(0, 0, 0x818480a, 0x81a0114, 0xbfb6c940)                                              = 1
--- SIGCHLD (Child exited) ---
 GC 13708 pts/1    S+     0:03      1    65  1742   648  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:35      1  1316 351871 347480 33.5 python3.2 Issue11849_test2.py
--- SIGCHLD (Child exited) ---
*** 13708 pts/1    S+     0:03      1    65  1742   648  0.0 ltrace -e malloc_trim python3.2 Issue11849_test2.py
13709 pts/1    S+     0:35      1  1316 351871 347480 33.5 python3.2 Issue11849_test2.py
+++ exited (status 0) +++
History
Date User Action Args
2011-04-19 02:41:39kaifengsetrecipients: + kaifeng, pitrou, flox, neologix
2011-04-19 02:41:39kaifengsetmessageid: <1303180899.72.0.186167652301.issue11849@psf.upfronthosting.co.za>
2011-04-19 02:41:37kaifenglinkissue11849 messages
2011-04-19 02:41:36kaifengcreate