classification
Title: Memory leaks in functions
Type: resource usage Stage: resolved
Components: Interpreter Core Versions: Python 3.8
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: inada.naoki, jdemeyer, pablogsal, serhiy.storchaka, twouters, vstinner
Priority: normal Keywords: patch

Created on 2018-05-03 17:54 by jdemeyer, last changed 2019-10-01 07:37 by nascheme. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 8058 merged inada.naoki, 2018-07-03 10:16
PR 15826 merged vstinner, 2019-09-10 07:41
PR 16502 merged nascheme, 2019-10-01 07:37
Messages (12)
msg316125 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2018-05-03 17:54
This is a memory leak:

>>> while True:
...    def f(): pass
...    f.__doc__ = f

This also:

>>> while True:
...     f = [].append                                                                                                                                                   
...     f.__module__ = f

This is because the classes "function" and "builtin_function_or_method" do not implement tp_clear.
msg320953 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-07-03 10:19
I'm not sure this should be backported to 3.6 and 2.7.
While this is a obvious bug, f.__module__ = f seems very unnatural.
msg320954 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2018-07-03 10:23
> While this is a obvious bug, f.__module__ = f seems very unnatural.

Sure.
msg321009 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2018-07-04 02:16
New changeset 3c452404ae178b742967589a0bb4a5ec768d76e0 by INADA Naoki in branch 'master':
bpo-33418: Add tp_clear for function object (GH-8058)
https://github.com/python/cpython/commit/3c452404ae178b742967589a0bb4a5ec768d76e0
msg350993 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 12:44
This change introduced the possibility to have function objects in an inconsistent state. For example, when calling tp_clear on the function some code must be invoked that tries to ca the function but some fields are NULL causing a crash.

I think we should revert this and think better how to protect against this situation.
msg350994 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 12:45
See also https://bugs.python.org/issue38006
msg351013 - (view) Author: Pablo Galindo Salgado (pablogsal) * (Python committer) Date: 2019-09-02 15:19
I am going to re-close this until we understand exactly how this is interacting with https://bugs.python.org/issue38006 as this seems more complicated than our first hypothesis.
msg351587 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-10 07:42
"f.__doc__ = f" or "f.__module__ = f" don't make any sense. It looks like a bug triggered on purpose. While it's bad, it's not a big deal.

The fix (commit 3c452404ae178b742967589a0bb4a5ec768d76e0) is causing way worse issues than the "artificial" memory leak (reference cycle). I wrote PR 15826 to revert the change in Python 3.8. bpo-38006 root issues are not well understood nor fixed yet.
msg351604 - (view) Author: Inada Naoki (inada.naoki) * (Python committer) Date: 2019-09-10 09:52
I'm OK to revert it in 3.8.

But I am worrying about func.func_closure.  Can it create cyclic reference in real life applications?
msg351605 - (view) Author: Thomas Wouters (twouters) * (Python committer) Date: 2019-09-10 09:57
New changeset ccaea525885e41c5f1e566bb68698847faaa82ca by T. Wouters (Victor Stinner) in branch '3.8':
Revert "bpo-33418: Add tp_clear for function object (GH-8058)" (GH-15826)
https://github.com/python/cpython/commit/ccaea525885e41c5f1e566bb68698847faaa82ca
msg351687 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2019-09-10 14:50
> But I am worrying about func.func_closure.  Can it create cyclic reference in real life applications?

remove.__closure__ is part of a reference cycle in bpo-38006.

I like func_clear() (which is still implemented in the master branch), but we need to understand why/how it was possible *crash Python* in bpo-38006.
msg351737 - (view) Author: Jeroen Demeyer (jdemeyer) * (Python triager) Date: 2019-09-10 18:58
> It looks like a bug triggered on purpose.

Absolutely. It's one of the many small issues that I found while working on PEP 590 and related things.
History
Date User Action Args
2019-10-01 07:37:55naschemesetpull_requests: + pull_request16095
2019-09-10 18:58:40jdemeyersetmessages: + msg351737
2019-09-10 14:50:52vstinnersetmessages: + msg351687
2019-09-10 09:57:34twouterssetnosy: + twouters
messages: + msg351605
2019-09-10 09:52:23inada.naokisetmessages: + msg351604
2019-09-10 07:42:59vstinnersetnosy: + vstinner
messages: + msg351587
2019-09-10 07:41:38vstinnersetpull_requests: + pull_request15474
2019-09-02 15:19:27pablogsalsetstatus: open -> closed
resolution: fixed
messages: + msg351013
2019-09-02 13:08:30pablogsalsetresolution: fixed -> (no value)
2019-09-02 12:45:38pablogsalsetnosy: inada.naoki, serhiy.storchaka, jdemeyer, pablogsal
resolution: fixed
messages: + msg350994
2019-09-02 12:44:28pablogsalsetstatus: closed -> open
resolution: fixed -> (no value)
2019-09-02 12:44:15pablogsalsetnosy: + pablogsal
messages: + msg350993
2018-07-04 02:16:14inada.naokisetstatus: open -> closed
stage: patch review -> resolved
resolution: fixed
versions: - Python 2.7, Python 3.6, Python 3.7
2018-07-04 02:16:00inada.naokisetmessages: + msg321009
2018-07-03 10:23:35jdemeyersetmessages: + msg320954
2018-07-03 10:19:55inada.naokisetnosy: + inada.naoki
messages: + msg320953
2018-07-03 10:16:42inada.naokisetkeywords: + patch
stage: patch review
pull_requests: + pull_request7667
2018-05-03 18:33:40serhiy.storchakasetnosy: + serhiy.storchaka

versions: + Python 2.7, Python 3.6, Python 3.7, Python 3.8
2018-05-03 17:54:46jdemeyersettype: resource usage
components: + Interpreter Core
2018-05-03 17:54:23jdemeyercreate