classification
Title: Reference leaks in Python 3.0rc3
Type: resource usage Stage: test needed
Components: Versions: Python 3.0
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: amaury.forgeotdarc, brett.cannon, christian.heimes, loewis
Priority: release blocker Keywords: patch

Created on 2008-11-21 02:28 by christian.heimes, last changed 2008-11-25 21:21 by christian.heimes. This issue is now closed.

Files
File name Uploaded Description Edit
issue4373_build_ext.patch christian.heimes, 2008-11-21 02:59
pickle-leak2.patch amaury.forgeotdarc, 2008-11-21 09:48
issue4373_build_ext2.patch christian.heimes, 2008-11-21 10:24
Messages (11)
msg76158 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-21 02:28
A refleak test of r67311 py3k shows several leaks:

test_distutils leaked [142, 142, 142, 142] references, sum=568
test_httpservers leaked [0, 0, 0, 217] references, sum=217
test_multiprocessing leaked [0, 0, 24, 0] references, sum=24
test_pickle leaked [1, 1, 1, 1] references, sum=4
test_pickletools leaked [1, 1, 1, 1] references, sum=4
test_telnetlib leaked [-84, 84, 0, 0] references, sum=0
test_threadedtempfile leaked [94, -94, 0, 0] references, sum=0
msg76159 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-21 02:40
Only distutils and pickle seem to have real leaks.

$ ./python Lib/test/regrtest.py -R:15: test_multiprocessing
test_multiprocessing
beginning 20 repetitions
12345678901234567890
....................
1 test OK.
[124096 refs]

$ ./python Lib/test/regrtest.py -R:15: test_distutils
test_distutils
beginning 20 repetitions
12345678901234567890
....................
test_distutils leaked [144, 144, 144, 144, 144, 144, 144, 144, 144, 144,
144, 144, 144, 144, 144] references, sum=2160
1 test OK.
[123507 refs]

$ ./python Lib/test/regrtest.py -R:15: test_httpservers
[...]
test_httpservers leaked [-217, 0, 198, 19, -35, 20, -202, 217, -217,
217, 0, 0, 0, 0, -33] references, sum=-33
1 test OK.
[96060 refs]

$ ./python Lib/test/regrtest.py -R:15: test_pickle
test_pickle
beginning 20 repetitions
12345678901234567890
....................
test_pickle leaked [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
references, sum=15
1 test OK.
[96783 refs]
msg76161 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-21 02:50
distutils.tests.test_build_ext leaks the references. I think it's
related to the xx module and totally harmless.
msg76162 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-21 02:59
This simple patch doesn't load the 'xx' module more than once in ref
leak tests.

The problem may also be caused by the xxmodule itself. Somebody should
give it a proper review :)
msg76167 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-21 09:07
I have studied this some time ago. The xx module comes each time from a
different file (in a directory created with tempfile.mkdtemp).

Every instance of the module creates a new entry in the static
"extensions" dictionary in Python/import.c. This entry at least contains
a copy of the module dictionary, which explains the number of "leaked"
references.

I do not see any way to clear this dictionary. The proposed patch is
probably the best thing to do.
msg76169 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-21 09:48
The refleak in test_pickle comes from unpickler_read(). The attached
patch corrects it.
msg76172 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-21 10:24
Good analysis Amaury! The new patch for build ext uses a single temp dir.
msg76175 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-21 10:57
The problem is that on Windows (and cygwin) you cannot unlink a .pyd
that is currently loaded in memory.

I tried to use ctypes and call FreeLibrary. Now the .pyd can be removed,
but the interpreter crashes when it comes to free the module on shutdown.
I'm afraid that until python has a real support for dlclose() on dynamic
loaded module, the best is to skip this test during the refleak hunt.
msg76422 - (view) Author: Brett Cannon (brett.cannon) * (Python committer) Date: 2008-11-25 19:33
Amaury's patch for pickle looks fine to me.
msg76428 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2008-11-25 21:12
Applied pickle-leak2.patch in 67381.
msg76431 - (view) Author: Christian Heimes (christian.heimes) * (Python committer) Date: 2008-11-25 21:21
Second fix applied in r67382
History
Date User Action Args
2008-11-25 21:21:59christian.heimessetstatus: open -> closed
resolution: fixed
messages: + msg76431
2008-11-25 21:12:36amaury.forgeotdarcsetmessages: + msg76428
2008-11-25 19:33:28brett.cannonsetnosy: + brett.cannon
messages: + msg76422
2008-11-21 15:17:43barrysetpriority: deferred blocker -> release blocker
2008-11-21 10:57:32amaury.forgeotdarcsetmessages: + msg76175
2008-11-21 10:24:34christian.heimessetfiles: + issue4373_build_ext2.patch
nosy: + loewis
messages: + msg76172
2008-11-21 09:48:16amaury.forgeotdarcsetfiles: + pickle-leak2.patch
messages: + msg76169
2008-11-21 09:07:48amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg76167
2008-11-21 02:59:18christian.heimessetfiles: + issue4373_build_ext.patch
keywords: + patch
messages: + msg76162
2008-11-21 02:50:07christian.heimessetmessages: + msg76161
2008-11-21 02:40:44christian.heimessetmessages: + msg76159
2008-11-21 02:28:58christian.heimescreate