classification
Title: test_io leaks memory
Type: resource usage Stage: resolved
Components: Extension Modules, Tests Versions: Python 3.1, Python 3.2, Python 2.7, Python 2.6
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: mark.dickinson, pitrou, ronaldoussoren
Priority: normal Keywords:

Created on 2010-08-01 15:53 by mark.dickinson, last changed 2010-08-01 17:02 by pitrou. This issue is now closed.

Messages (6)
msg112348 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-01 15:53
regrest -L detects a memory leak in test_io, on OS X 10.6.4.


newton:py3k dickinsm$ ./python.exe Lib/test/regrtest.py -L test_io
test_io

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.

Testing large file ops skipped on darwin.
It requires 2147483648 bytes and a long time.
Use 'regrtest.py -u largefile test_io' to run it.
1 test OK.
leaks Report Version:  2.0
Process:         python.exe [71362]
Path:            /Users/dickinsm/python/svn/py3k/python.exe
Load Address:    0x100000000
Identifier:      python.exe
Version:         ??? (???)
Code Type:       X86-64 (Native)
Parent Process:  bash [18674]

Date/Time:       2010-08-01 16:51:50.332 +0100
OS Version:      Mac OS X 10.6.4 (10F569)
Report Version:  6

Process 71362: 6679 nodes malloced for 14021 KB
Process 71362: 16 leaks for 2048 total leaked bytes.
Leak: 0x10138bb40  size=128  zone: DefaultMallocZone_0x1002f7000	
	0x00000000 0x00000000 0x434f4e44 0x00000000 	........DNOC....
	0x00000000 0x80000000 0x00000000 0x00000000 	................
	0x00000000 0x00000000 0x00000000 0x00000000 	................
	0x00000000 0x00000000 0x4d555458 0x00000000 	........XTUM....
	0x00000000 0x00000060 0x00000000 0x00000000 	....`...........
	0x00000000 0x00000000 0x00000000 0x00000000 	................
	0x00000000 0x00000000 0x00000000 0x00000000 	................
	0x00000000 0x00000000 0xdbdbdbdb 0x0008dbdb 	................

(15 blocks of similar output omitted)

I'm investigating further...
msg112349 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-01 16:10
test_constructor (test.test_io.CBufferedReaderTest)

appears to be one of the offending tests.

This is probably not an OS X specific problem.  Removing 'Mac' from components.
msg112350 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-01 16:39
Minimal example to reproduce:  the script below exhausts my system memory in a minute or two.

import io
rawio = io.BytesIO(b"abc")
bufio = io.BufferedReader(rawio)
while True:
    bufio.__init__(rawio)
msg112351 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-01 16:40
Can you try the following patch:

diff -r 962e1a7a40fd Modules/_io/bufferedio.c
--- a/Modules/_io/bufferedio.c	Sun Aug 01 17:30:56 2010 +0200
+++ b/Modules/_io/bufferedio.c	Sun Aug 01 18:39:39 2010 +0200
@@ -636,6 +636,8 @@ _buffered_init(buffered *self)
         return -1;
     }
 #ifdef WITH_THREAD
+    if (self->lock)
+        PyThread_free_lock(self->lock);
     self->lock = PyThread_allocate_lock();
     if (self->lock == NULL) {
         PyErr_SetString(PyExc_RuntimeError, "can't allocate read lock");
msg112354 - (view) Author: Mark Dickinson (mark.dickinson) * (Python committer) Date: 2010-08-01 16:47
Thanks!  That fixes all the test_io leaks I was seeing.
msg112358 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2010-08-01 17:02
Thanks. Fixed in r83411 (py3k), r83412 (2.7), r83413 (3.1).
History
Date User Action Args
2010-08-01 17:02:06pitrousetstatus: open -> closed
resolution: fixed
messages: + msg112358

stage: needs patch -> resolved
2010-08-01 16:47:50mark.dickinsonsetmessages: + msg112354
2010-08-01 16:40:22pitrousettype: behavior -> resource usage
versions: + Python 2.6, Python 3.1, Python 2.7
2010-08-01 16:40:04pitrousetnosy: + pitrou
messages: + msg112351
2010-08-01 16:39:27mark.dickinsonsetmessages: + msg112350
2010-08-01 16:10:12mark.dickinsonsetassignee: ronaldoussoren ->
messages: + msg112349
components: - macOS
nosy: ronaldoussoren, mark.dickinson
2010-08-01 15:53:14mark.dickinsoncreate