classification
Title: refleak in C functools.lru_cache
Type: resource usage Stage: resolved
Components: Library (Lib) Versions: Python 3.7, Python 3.6, Python 3.5
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: yselivanov Nosy List: gvanrossum, larry, levkivskyi, ned.deily, python-dev, serhiy.storchaka, yselivanov
Priority: release blocker Keywords: patch

Created on 2016-11-09 23:47 by yselivanov, last changed 2016-11-10 01:11 by yselivanov. This issue is now closed.

Files
File name Uploaded Description Edit
refleak.patch yselivanov, 2016-11-09 23:47 review
Messages (12)
msg280468 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-11-09 23:47
The attached patch fixes a refleak in functools.lru_cache.
msg280470 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2016-11-09 23:49
LGTM.
msg280471 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-11-09 23:51
FWIW, if I comment out this code

try:
    from _functools import _lru_cache_wrapper
except ImportError:
    pass

in functools.py to use the pure Python version, then I get much larger numbers:

$ ./python -m test -R : test_typing
Run tests sequentially
0:00:00 [1/1] test_typing
beginning 9 repetitions
123456789
.........
test_typing leaked [25003, 25003, 25003, 25003] references, sum=100012
test_typing leaked [9350, 9352, 9352, 9352] memory blocks, sum=37406
test_typing failed
msg280472 - (view) Author: Roundup Robot (python-dev) (Python triager) Date: 2016-11-09 23:57
New changeset ba59f3328032 by Yury Selivanov in branch '3.5':
Issue #28653: Fix a refleak in functools.lru_cache.
https://hg.python.org/cpython/rev/ba59f3328032

New changeset 5b253d641826 by Yury Selivanov in branch '3.6':
Merge 3.6 (issue #28653)
https://hg.python.org/cpython/rev/5b253d641826

New changeset 784fea019cab by Yury Selivanov in branch 'default':
Merge 3.6 (issue #28653)
https://hg.python.org/cpython/rev/784fea019cab
msg280473 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-11-10 00:03
> FWIW, if I comment out this code

Oh boy... This is something else. Please re-open #28649 explaining that it now leaks with pure-Python lru_cache.  Could you please take a look at it?
msg280474 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-11-10 00:10
It seems to be unrelated to typing.py
With your test from test_functools:

    def test_lru_type_error(self):
        @functools.lru_cache(maxsize=None)
        def infinite_cache(o):
            pass
        with self.assertRaises(TypeError):
            infinite_cache([])

I get [2, 2, 2, 2] for unpatched C version (this is now fixed)
and [24764, 24764, 24764, 24764] for Python version.

Should I maybe open a separate issue for this?
msg280475 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-11-10 00:13
Anyway, this is not urgent, typing.py uses the default version, which is the C version.
msg280476 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-11-10 00:13
> It seems to be unrelated to typing.py

Wow. I can reproduce refleaks in test_typing, but not in test_functools. This is weird.
msg280477 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-11-10 00:17
> Anyway, this is not urgent

Not so sure, a refleak in pure-python version kind of scares me.  It can be a bug in the core.
msg280478 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-11-10 00:26
> Wow. I can reproduce refleaks in test_typing, but not in test_functools. This is weird.

Indeed, I copied the new test from test_functools to test_typing, but now I see that pure Python version of lru_cache fails even if I remove this test and test that was previously failing with C version (test_extended_generic_rules_eq).

> Not so sure, a refleak in pure-python version kind of scares me.  It can be a bug in the core.

Indeed. Now we need to figure out what it so special about typing that triggers the leak.
msg280479 - (view) Author: Ivan Levkivskyi (levkivskyi) * (Python committer) Date: 2016-11-10 00:43
I am not sure what does this mean, but when I remove __slots__ = () from some classes, numbers of leaked references change.
msg280480 - (view) Author: Yury Selivanov (yselivanov) * (Python committer) Date: 2016-11-10 01:11
Ivan, I think I figured it out, I'll close this one and reopen the typing one.
History
Date User Action Args
2016-11-10 01:11:57yselivanovsetstatus: open -> closed
title: refleak in functools.lru_cache -> refleak in C functools.lru_cache
messages: + msg280480

resolution: fixed
stage: needs patch -> resolved
2016-11-10 00:43:18levkivskyisetmessages: + msg280479
2016-11-10 00:26:52levkivskyisetmessages: + msg280478
2016-11-10 00:17:26yselivanovsetmessages: + msg280477
2016-11-10 00:14:26yselivanovsetstage: commit review -> needs patch
2016-11-10 00:13:35yselivanovsetmessages: + msg280476
2016-11-10 00:13:16levkivskyisetmessages: + msg280475
2016-11-10 00:10:29levkivskyisetmessages: + msg280474
2016-11-10 00:03:00yselivanovsetmessages: + msg280473
2016-11-09 23:57:17python-devsetnosy: + python-dev
messages: + msg280472
2016-11-09 23:51:37levkivskyisetnosy: + levkivskyi
messages: + msg280471
2016-11-09 23:49:34serhiy.storchakasetassignee: yselivanov
messages: + msg280470
stage: patch review -> commit review
2016-11-09 23:47:28yselivanovcreate