classification
Title: test.regrtest: use tracemalloc to detect memory leaks?
Type: Stage:
Components: Tests Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder:
Assigned To: Nosy List: Saimadhav.Heblikar, ncoghlan, neologix, pitrou, vstinner, zach.ware
Priority: normal Keywords: patch

Created on 2013-11-27 17:49 by vstinner, last changed 2015-03-18 12:44 by vstinner. This issue is now closed.

Files
File name Uploaded Description Edit
regrtest.patch vstinner, 2014-06-12 12:48
Messages (11)
msg204604 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2013-11-27 17:49
Someone proposed to use the new tracemalloc module in test.regrtest, I don't remember who. Maybe Nick Coghlan?

I tried an hack dash_R(), but the result was not reliable. I should try harder :-)
msg220343 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-12 12:48
I tried different options to show the memory leaks found by tracemalloc, but I'm not satisfied by any option. I get a lot of noise, the output is almost useless. Randomly, between 1 and 1000 KB are allocated or released in random files: in unittest or linecache modules for example.

It looks like the major issue is that the unittest leaks memory. For example, there are still 4 TestCase instances alive after the execution of test_sys. But later, these instances are deleted.

It looks like it creates reference leaks and so memory is only released late. Calling gc.collect() doesn't help. I remember that I already saw something strange with the _Outcome class used in unittest.TestCase.run(). See the issue #19880 (changeset 09658ea0b93d).

The asyncio module has a similar issue: it stores an exception which indirectly contains a reference cycle in the Exception.__traceback__ attribute.
http://bugs.python.org/issue17911
https://code.google.com/p/tulip/issues/detail?id=155
msg220344 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-06-12 12:50
regrtest.patch is a work-in-progress patch. It shows the "top 10" when the -R option is used, ex: "python -m test -R 3:3 test_sys".
msg222697 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-10 20:28
The problem is snapshots will by themselves create allocation noise: you cannot really use tracemalloc to detect leaks, only to diagnose the leaks you have detected.
msg222698 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-10 20:30
> The problem is snapshots will by themselves create allocation noise: you cannot really use tracemalloc to detect leaks, only to diagnose the leaks you have detected.

It's trivial to ignore allocations done in the tracemalloc module. My work-in-progress patch uses for example these filters:

tracemalloc_filters = [
   tracemalloc.Filter(False, '<frozen importlib._bootstrap>', all_frames=True),
   tracemalloc.Filter(False, tracemalloc.__file__, all_frames=True),
]
msg222701 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-07-10 20:55
> My work-in-progress patch uses for example these filters

If you use filters, you need to add filters for fnmatch and a couple other things :)
msg222702 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-07-10 21:15
2014-07-10 22:55 GMT+02:00 Antoine Pitrou <report@bugs.python.org>:
> If you use filters, you need to add filters for fnmatch and a couple other things :)

tracemalloc.Filter(False, tracemalloc.__file__, all_frames=True)
ignores all memory allocated directly or indirectly by the tracemalloc
module: notice the all_frames=True parameter. I'm saving at least 5
frames in tracemalloc when I work on this issue.
msg225266 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-08-13 07:49
Some of the changes to the way regrtest handles test cases has shown a weird cycle in test_codecs - see issue #22166 for details.

I'll try Victor's patch to see if it's more enlightening than my hacked together attempt.
msg225267 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2014-08-13 07:55
On second thoughts, what Victor is doing isn't that different from my existing hack - I'll lift the display code, but I'm happy with my data collection code.

One thing I did that seemed to reduce the noise substantially was to leave tracing turned off most of the time, and have a start/stop pair around the actual execution of the test and the subsequent cache cleanup.
msg225277 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-08-13 13:25
Slightly unrelated, but in numba I've rewritten parallel testing and refleak testing as regular unittest extensions (TestRunner / TestResult subclasses). It also means you have per-test method refleak results, instead of per-test file. See e.g. RefleakTestResult in https://github.com/numba/numba/blob/master/numba/tests/__init__.py
msg238421 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2015-03-18 12:44
I'm no more interested to work on this issue. regrtest.patch is unstable, and I don't know how to make it more reliable. Antoine's allocation counter is enough right now.
History
Date User Action Args
2015-03-18 12:44:24vstinnersetstatus: open -> closed
resolution: out of date
messages: + msg238421
2014-08-13 13:25:51pitrousetnosy: + zach.ware
messages: + msg225277
2014-08-13 07:55:42ncoghlansetmessages: + msg225267
2014-08-13 07:49:15ncoghlansetmessages: + msg225266
2014-08-13 07:47:09ncoghlanlinkissue22190 superseder
2014-07-10 21:15:05vstinnersetmessages: + msg222702
2014-07-10 20:55:25pitrousetmessages: + msg222701
2014-07-10 20:30:56vstinnersetmessages: + msg222698
2014-07-10 20:28:52pitrousetnosy: + pitrou
messages: + msg222697
2014-06-12 12:50:20vstinnersetnosy: + Saimadhav.Heblikar
messages: + msg220344
2014-06-12 12:48:40vstinnersetfiles: + regrtest.patch
keywords: + patch
messages: + msg220343
2013-11-27 17:49:27vstinnercreate