classification
Title: memory leaks in py3k
Type: resource usage Stage: resolved
Components: Extension Modules, Interpreter Core Versions: Python 3.2, Python 2.7
process
Status: closed Resolution: fixed
Dependencies: 5990 6002 Superseder:
Assigned To: Nosy List: alexandre.vassalotti, asvetlov, flox, ocean-city, orsenthil, pitrou
Priority: critical Keywords: patch

Created on 2009-03-29 14:19 by pitrou, last changed 2010-02-09 14:36 by flox. This issue is now closed.

Files
File name Uploaded Description Edit
issue5596_urllib_py3k.diff flox, 2009-12-07 23:23 Patch, apply to py3k
issue5596_pipes_py3k.diff flox, 2009-12-07 23:24 Patch, apply to py3k
issue5596_urllib.diff flox, 2009-12-07 23:44 Patch, apply to trunk
issue5596_zipimport_support.diff flox, 2009-12-08 01:22 Patch, apply to trunk
issue5596_zipimport_support_py3k.diff flox, 2009-12-08 01:23 Patch, apply to py3k
Messages (25)
msg84403 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 14:19
A couple of tests exhibit some memory leaks in the py3k branch (when
invoked with "-R 3:2").

test_asyncore leaked [-78, 0] references, sum=-78
test_fileio leaked [1, 1] references, sum=2
test_httpservers leaked [-210, 157] references, sum=-53
test_socket leaked [8, 210] references, sum=218
test_urllib2 leaked [227, 227] references, sum=454
test_urllib2_localnet leaked [5, 4] references, sum=9

I'm filing this as critical so that we keep track of it, but it doesn't
look severe in itself (ISTR we actually had more leaks in 3.0).
msg84421 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-29 18:33
I hope attached patch will fix test_fileio leak.
msg84422 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-29 18:38
I hope attached patch will fix leak of test_socket.py.

Leak is happening in FileObjectClassTestCase#testFullRead and
testCloseAfterMakefile. (actually, IOBase#readall)
msg84423 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 18:41
Applied in r70690, thanks!
msg84437 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-03-29 22:14
The only remaining leaks now are:

test_urllib2 leaked [227, 227] references, sum=454
test_urllib2_localnet leaked [3, 3] references, sum=6
msg84529 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-03-30 08:10
About test_urllib2_localnet, this leak happens because FakeProxyHandler
has class variable digest_auth_handler and its instance variables are
modified. Following workaround can fix this leak, but I'm not sure this
can be committed.

Index: Lib/test/test_urllib2_localnet.py
===================================================================
--- Lib/test/test_urllib2_localnet.py	(revision 70694)
+++ Lib/test/test_urllib2_localnet.py	(working copy)
@@ -241,6 +241,7 @@
 
     def tearDown(self):
         self.server.stop()
+        FakeProxyHandler.digest_auth_handler.__init__()
 
     def test_proxy_with_bad_password_raises_httperror(self):
         self._digest_auth_handler.add_password(self.REALM, self.URL,
msg85449 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-04-05 00:03
I have the following tests leaking on my Linux build of py3k:


test_structmembers leaked [102, 102, 102] references, sum=306
test_structmembers leaked [102, 102, 102] references, sum=306
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [17, 17, 17, 17] references, sum=68
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102] references, sum=408
test_structmembers leaked [102, 102, 102, 102, 102, 102] references, sum=612
test_structmembers leaked [106, 102, 102] references, sum=310
test_structmembers leaked [102, 102, 102] references, sum=306
test_structmembers leaked [106, 102, 102] references, sum=310
test_structmembers leaked [106, 102, 102] references, sum=310
test_docxmlrpc leaked [166, -251, 0] references, sum=-85
test_httpservers leaked [-195, 0, 0] references, sum=-195
test_os leaked [0, 0, 23] references, sum=23
test_popen leaked [-23, 23, -23] references, sum=-23
test_telnetlib leaked [0, 0, 84] references, sum=84
test_threading leaked [-119, 119, -28] references, sum=-28
test_urllib2 leaked [227, 227, 227] references, sum=681
test_urllib2_localnet leaked [-251, 3, 3] references, sum=-245
msg85452 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-04-05 00:07
Strange, I don't see any leaks in test_structmembers here.

test_structmembers
beginning 9 repetitions
123456789
.........
1 test OK.
[89785 refs]

This is with r71202 on a 64-bit build.
msg85457 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-04-05 01:13
Dismiss my last comment. These are the tests actually leaking for my build: 

test_docxmlrpc leaked [85, -6, 0, 39] references, sum=118
test_httpservers leaked [-195, 195, 0, 64] references, sum=64
test_os leaked [23, -23, 0, 23] references, sum=23
test_pipes leaked [0, 0, 0, -23] references, sum=-23
test_popen leaked [0, -23, 0, 0] references, sum=-23
test_urllib2 leaked [227, 227, 227, 227] references, sum=908
test_urllib2_localnet leaked [3, 3, 3, 3] references, sum=12
msg86915 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-01 21:39
Current status here:

test_multiprocessing leaked [24, 0] references, sum=24
test_urllib2 leaked [227, 227] references, sum=454
test_urllib2_localnet leaked [3, 3] references, sum=6
msg87542 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-10 20:21
New results:

test_os leaked [1, 1] references, sum=2
test_urllib leaked [4, 2] references, sum=6
test_urllib2 leaked [227, 227] references, sum=454
test_urllib2_localnet leaked [3, 3] references, sum=6
msg88267 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-24 16:43
Current results:

test_modulefinder leaked [145, 145] references, sum=290
test_urllib leaked [10, 0] references, sum=10
test_urllib2 leaked [227, 227] references, sum=454
msg89295 - (view) Author: Alexandre Vassalotti (alexandre.vassalotti) * (Python committer) Date: 2009-06-12 19:04
These tests are leaking on my build of r73393:

test_docxmlrpc leaked [0, 0, 85, 0, 0, -85, 0, 0, 0, 0] references, sum=0
test_popen leaked [0, 23, -23, 0, 23, 0, -23, 23, -23, 23] references,
sum=23
test_urllib leaked [4, 4, 4, 4, 6, 0, 0, 0, 2, 0] references, sum=24
test_urllib2_localnet leaked [0, 0, 0, 0, 0, 0, 251, -250, 0, 0]
references, sum=1
test_os leaked [0, -26, 26, 0, -26, 26, 0, 23, 0, 0] references, sum=23
msg90766 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-07-21 18:26
With the py3k branch head (r74140):

test_urllib leaked [6, 4] references, sum=10
test_urllib2 leaked [227, 227] references, sum=454
msg93991 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-14 14:20
With r75397:

test___all__ leaked [1, 1] references, sum=2
test_asyncore leaked [1, 0] references, sum=1
test_distutils leaked [0, 2] references, sum=2
test_httpservers leaked [-259, 0] references, sum=-259
test_os leaked [-23, 23] references, sum=0
test_pydoc leaked [0, -21] references, sum=-21
test_threaded_import leaked [1, 0] references, sum=1
test_urllib leaked [4, 4] references, sum=8
test_uuid leaked [6, 6] references, sum=12
test_urllib2_localnet leaked [240, -239] references, sum=1
msg94568 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-27 18:09
With r75820:

test_asyncore leaked [153, -152] references, sum=1
test_distutils leaked [0, 2] references, sum=2
test_httpservers leaked [0, 259] references, sum=259
test_pipes leaked [23, 0] references, sum=23
test_threaded_import leaked [1, 0] references, sum=1
test_tokenize leaked [0, 291] references, sum=291
test_urllib leaked [6, 4] references, sum=10
test_uuid leaked [6, 6] references, sum=12
test_zipimport_support leaked [748, 0] references, sum=748
test_zipfile leaked [2, 0] references, sum=2
msg94574 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-27 19:06
test_asyncore and test_httpservers should now be fixed.
msg94739 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-10-30 23:48
After quite a bit of work, here is the new status as of r75970:

test_textwrap leaked [0, -206] references, sum=-206
test_urllib leaked [2, 0] references, sum=2
msg94740 - (view) Author: R. David Murray (r.david.murray) * (Python committer) Date: 2009-10-31 01:01
And there was much rejoicing :)
msg96051 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-07 14:31
test_textwrap seems OK with r76697

I still have leaks on test_pipes and test_urllib:

~ $ ./python Lib/test/regrtest.py -R : test_textwrap \
             test_pipes test_urllib
test_pipes leaked [0, -26, 0, 26] references, sum=0
test_pipes leaked [0, -26, 26, -26] references, sum=-26
test_pipes leaked [-26, 26, 0, 0] references, sum=0
test_pipes leaked [26, -26, 26, 0] references, sum=26
test_urllib leaked [0, 2, 0, 0] references, sum=2
test_urllib leaked [4, 6, 0, 2] references, sum=12
test_urllib leaked [2, 4, 2, 0] references, sum=8
test_urllib leaked [6, 6, 2, 0] references, sum=14
msg96084 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-07 23:26
Patch for the test refleaks:
 * test_pipes
 * test_urllib
msg96090 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-07 23:44
And the test_urllib may leak on 2.7, too.

Patch attached.
msg96101 - (view) Author: Florent Xicluna (flox) * (Python committer) Date: 2009-12-08 01:22
Another false-positive in test_zipimport_support.
Both on 2.7 and 3.2.

Patch attached.
msg96150 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-08 19:45
I've committed fixes to test_urllib and test_zipimport_support similar
to the ones you proposed. Thank you!
msg96159 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-12-08 23:48
No leaks anymore!
http://mail.python.org/pipermail/python-checkins/2009-December/088162.html
Thanks flox!
History
Date User Action Args
2010-02-09 14:36:18floxsetstatus: open -> closed
2009-12-13 18:11:59pitrousetresolution: fixed
stage: needs patch -> resolved
2009-12-08 23:48:43pitrousetmessages: + msg96159
2009-12-08 19:45:40pitrousetmessages: + msg96150
2009-12-08 02:25:49orsenthilsetnosy: + orsenthil
2009-12-08 02:22:19r.david.murraysetnosy: - r.david.murray
2009-12-08 01:23:13floxsetfiles: + issue5596_zipimport_support_py3k.diff
2009-12-08 01:22:10floxsetfiles: + issue5596_zipimport_support.diff

messages: + msg96101
2009-12-08 00:20:45floxsetversions: + Python 2.7
2009-12-07 23:44:39floxsetfiles: + issue5596_urllib.diff

messages: + msg96090
2009-12-07 23:26:30floxsetmessages: + msg96084
2009-12-07 23:24:08floxsetfiles: + issue5596_pipes_py3k.diff
2009-12-07 23:23:43floxsetfiles: + issue5596_urllib_py3k.diff
2009-12-07 14:31:29floxsetnosy: + flox
messages: + msg96051
2009-12-06 04:42:51asvetlovsetnosy: + asvetlov
2009-10-31 03:32:19ocean-citylinkissue1739118 superseder
2009-10-31 01:01:48r.david.murraysetmessages: + msg94740
2009-10-30 23:48:14pitrousetmessages: + msg94739
2009-10-27 19:06:33pitrousetmessages: + msg94574
2009-10-27 18:09:45pitrousetmessages: + msg94568
2009-10-14 14:20:33pitrousetmessages: + msg93991
2009-07-21 18:26:08pitrousettitle: memory leaks in 3.1 -> memory leaks in py3k
messages: + msg90766
versions: + Python 3.2, - Python 3.1
2009-06-12 19:04:24alexandre.vassalottisetmessages: + msg89295
2009-05-29 01:01:33r.david.murraysetnosy: + r.david.murray
2009-05-24 16:43:39pitrousetmessages: + msg88267
2009-05-11 22:35:19r.david.murraysetdependencies: + test_urllib2_localnet DigestAuthHandler leaks nonces
2009-05-10 20:45:25pitrousetdependencies: + Memory leak in os.rename() and other functions
2009-05-10 20:21:02pitrousetmessages: + msg87542
2009-05-01 21:39:20pitrousetfiles: - py3k_fix_leak_of_test_socket.patch
2009-05-01 21:39:17pitrousetfiles: - py3k_fix_leak_of_fileio.patch
2009-05-01 21:39:12pitrousetmessages: + msg86915
2009-04-05 01:13:50alexandre.vassalottisetmessages: + msg85457
2009-04-05 00:07:55pitrousetmessages: + msg85452
2009-04-05 00:03:36alexandre.vassalottisetnosy: + alexandre.vassalotti
messages: + msg85449
2009-03-30 08:10:59ocean-citysetmessages: + msg84529
2009-03-29 22:14:00pitrousetmessages: + msg84437
2009-03-29 18:41:32pitrousetmessages: + msg84423
2009-03-29 18:38:20ocean-citysetfiles: + py3k_fix_leak_of_test_socket.patch

messages: + msg84422
2009-03-29 18:33:45ocean-citysetfiles: + py3k_fix_leak_of_fileio.patch

nosy: + ocean-city
messages: + msg84421

keywords: + patch
2009-03-29 14:19:47pitroucreate