classification
Title: Add gc.needs_finalizing() to check if an object needs finalising
Type: enhancement Stage: needs patch
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: out of date
Dependencies: Superseder: Generator cleanup without tp_del
View: 17807
Assigned To: Nosy List: alex, barry, benjamin.peterson, fijall, isoschiz, ncoghlan, pitrou, sbt
Priority: low Keywords: easy

Created on 2013-04-20 06:38 by ncoghlan, last changed 2015-04-14 23:59 by benjamin.peterson. This issue is now closed.

Messages (8)
msg187409 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-20 06:38
This came up in issue 17468: currently, populating tp_del from C (as generators now do) doesn't automatically create a __del__ wrapper visible from Python.

The rationale given in the initial commit is that there's no need to define a wrapper, since tp_del won't be populated from C code (that will use tp_dealloc instead), but there's now at least one case where it *is* populated from C (generators), which means it behaves *as if* __del__ is defined (since the interpreter actually checks the tp_del slot), but *looks* like __del__ *isn't* defined (since there is no wrapper created).

Independent of the memory leak concerns with generators defining tp_del, it would be better if a wrapper function was defined so the existence of the method was at least visible from Python code.
msg187412 - (view) Author: Richard Oudkerk (sbt) * (Python committer) Date: 2013-04-20 09:44
Would this mean that the destructor could be run more than once (or prematurely)?
msg187417 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2013-04-20 11:47
Sounds reasonable to me. Note that it won't remove the special-casing in gcmodule.c:

static int
has_finalizer(PyObject *op)
{
    if (PyGen_CheckExact(op))
        return PyGen_NeedsFinalizing((PyGenObject *)op);
    else
        return op->ob_type->tp_del != NULL;
}
msg187428 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-04-20 14:27
What exactly would calling such a wrapper do?
msg187435 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-20 15:27
Calling __del__ explicitly shouldn't be any worse than doing the same thing for any other type implemented in Python (or, in the case of generators, calling close() multiple times). What I'm mostly interested in is the "can this type cause uncollectable cycles" introspection aspect.

However, as Antoine noted, generators are an interesting special case because the GC is able to *skip* finalising them in some cases, so exposing __del__ isn't right for them either (as that suggests they will *always* be uncollectable in a cycle, when that isn't the case).

So now I'm wondering if a better answer may be to generalise the current generator special case to a "__needsdel__" protocol: provide a __del__ method, but always make it possible for the GC to skip it when it wouldn't do anything (e.g. if you've already called close() explicitly). PyGenerator_NeedsFinalizing would then become the __needsdel__ impl for generators, and we could lose the special casing in the GC code. From Python, you could detect the three cases through:

__del__ only: can cause uncollectable cycles
__del__and __needsdel__: can cause uncollectable cycles, but it depends on the instance state
Neither: can't cause uncollectable cycles
msg187436 - (view) Author: Benjamin Peterson (benjamin.peterson) * (Python committer) Date: 2013-04-20 15:44
I don't understand why we need to invent a protocol for this. The gc module already has methods and members for introspecting the collection. I don't think the gen special casing currently needs to be generalized. (What would use it?)
msg187438 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-20 16:01
Yeah, I've figured out that rather than exposing __del__ if tp_del is populated, or generalising the generator special case, the simplest way to make this info accessible is to be able to ask the *garbage collector* if it thinks an object needs finalising.

That actually makes this a pretty easy issue (as C issues go) - it's just a matter of exposing http://hg.python.org/cpython/file/default/Modules/gcmodule.c#l525 (has_finalizer) as gc.needs_finalizing.

It will be easier once #17468 is done though, since that will make it clearer what the documentation should say.
msg187493 - (view) Author: Nick Coghlan (ncoghlan) * (Python committer) Date: 2013-04-21 04:48
Antoine came up with a scheme (see issue 17807) that should eliminate the tp_del implementation from generator objects by moving the cleanup code to the frame deallocation.

This will restore Guido's original assumption that types implemented in C won't need to provide tp_del, so this proposal becomes obsolete.
History
Date User Action Args
2015-04-14 23:59:24benjamin.petersonsetstatus: open -> closed
2013-04-21 04:50:36alexsetnosy: + alex, fijall
2013-04-21 04:48:14ncoghlansetsuperseder: Generator cleanup without tp_del
dependencies: - Generator memory leak
resolution: out of date
messages: + msg187493
2013-04-20 16:01:18ncoghlansetkeywords: + easy

dependencies: + Generator memory leak
messages: + msg187438
title: Expose __del__ when tp_del is populated from C code -> Add gc.needs_finalizing() to check if an object needs finalising
2013-04-20 15:44:12benjamin.petersonsetmessages: + msg187436
2013-04-20 15:27:33ncoghlansetmessages: + msg187435
2013-04-20 14:47:58barrysetnosy: + barry
2013-04-20 14:29:43isoschizsetnosy: + isoschiz
2013-04-20 14:27:32benjamin.petersonsetnosy: + benjamin.peterson
messages: + msg187428
2013-04-20 11:47:24pitrousetnosy: + pitrou
messages: + msg187417
2013-04-20 09:44:35sbtsetnosy: + sbt
messages: + msg187412
2013-04-20 06:38:21ncoghlancreate