classification
Title: Investigated ref leak report related to thread regrtest.py
Type: resource usage Stage: patch review
Components: Interpreter Core Versions: Python 2.6
process
Status: closed Resolution: duplicate
Dependencies: Superseder: memory leaks in py3k
View: 5596
Assigned To: Nosy List: ajaksu2, ocean-city, pitrou
Priority: normal Keywords: patch

Created on 2007-06-18 14:43 by ocean-city, last changed 2009-10-31 03:32 by ocean-city. This issue is now closed.

Files
File name Uploaded Description Edit
archive.zip ocean-city, 2007-06-18 14:43
archive.zip ocean-city, 2007-06-18 17:48 version2
test_leak.py ajaksu2, 2009-04-06 09:39 Test thread-related ref leaks
experimental.patch ajaksu2, 2009-04-06 09:41 Help diagnosing thread-related ref leaks
Messages (6)
msg32364 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2007-06-18 14:43
Hello. I investigated ref leak report related to thread.
Please run python regrtest.py -R :: test_leak.py (attached file)
Sometimes ref leak is reported.
# I saw this as regression failure on python-checkins.

# total ref count 92578 -> 92669
  _Condition 2
  Thread 6
  _Event 1
  bool 10
  instancemethod 1
  code 2
  dict 9
  file 1
  frame 3
  function 2
  int 1
  list 2
  builtin_function_or_method 5
  NoneType 2
  str 27
  thread.lock 7
  tuple 5
  type 5

Probably this happens because threading.Thread is implemented as Python
code,
(expecially threading.Thread#join), the code of regrtest.py

        if i >= nwarmup:
            deltas.append(sys.gettotalrefcount() - rc - 2)

can run before thread really quits. (before Moudles/threadmodule.c
t_bootstrap()'s

 Py_DECREF(boot->func);
 Py_DECREF(boot->args);
 Py_XDECREF(boot->keyw);

runs)

So I experimentally inserted the code to wait for thread termination.
(attached file experimental.patch) And I confirmed error was gone.

# Sorry for hackish patch which only runs on windows. It should run
# on other platforms if you replace Sleep() in Python/sysmodule.c
# sys_debug_ref_leak_leave() with appropriate function.
msg32365 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2007-06-18 17:48
Sorry, I updated the patch. Moved

fprintf(stderr, "================= thread leave\n");
current_thread_count--;

in threadmodule.c just before PyThread_exit_thread().
File Added: archive.zip
msg32366 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2007-06-19 01:13
Umm, maybe still not enough? applied this patch (fix for 3 str leaks) test_urllib2_localnet.py  still reports leaks sometimes.

Index: Lib/test/test_urllib2_localnet.py
===================================================================
--- Lib/test/test_urllib2_localnet.py	(revision 56002)
+++ Lib/test/test_urllib2_localnet.py	(working copy)
@@ -208,7 +208,7 @@
     testing.
     """
 
-    digest_auth_handler = DigestAuthHandler()
+    digest_auth_handler = None
 
     def log_message(self, format, *args):
         # Uncomment the next line for debugging.
@@ -240,6 +240,7 @@
     PROXY_URL = "http://127.0.0.1:%d" % PORT
 
     def setUp(self):
+        FakeProxyHandler.digest_auth_handler = DigestAuthHandler()
         FakeProxyHandler.digest_auth_handler.set_users({
             self.USER : self.PASSWD
             })
@@ -257,6 +258,7 @@
 
     def tearDown(self):
         self.server.stop()
+        FakeProxyHandler.digest_auth_handler = None
 
     def test_proxy_with_bad_password_raises_httperror(self):
         self._digest_auth_handler.add_password(self.REALM, self.URL,

///////////////////////////

test_urllib2_localnet leaked [2, 0, 0, 0] references, sum=2

# it is now rare case though....
msg85622 - (view) Author: Daniel Diniz (ajaksu2) * (Python triager) Date: 2009-04-06 09:39
Attaching Hirokazu's tests and experimental patch as plaintext.
msg87547 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2009-05-10 21:01
Hmm, rather than some C code in the internals, couldn't we simply
iterate over the registered Thread objects and join() on those which are
still running?
msg94741 - (view) Author: Hirokazu Yamamoto (ocean-city) * (Python committer) Date: 2009-10-31 03:32
This issue seems to be fixed in r75958.
History
Date User Action Args
2009-10-31 03:32:20ocean-citysetstatus: open -> closed
resolution: duplicate
superseder: memory leaks in py3k
messages: + msg94741
2009-05-10 21:01:50pitrousetnosy: + pitrou
messages: + msg87547
2009-04-06 09:41:31ajaksu2setfiles: + experimental.patch
title: Investigated ref leak report related to thread(regrtest.py - -> Investigated ref leak report related to thread regrtest.py
2009-04-06 09:39:55ajaksu2setfiles: + test_leak.py

type: resource usage

keywords: + patch
nosy: + ajaksu2
messages: + msg85622
stage: patch review
2007-06-18 14:43:50ocean-citycreate