Title: __del__() order is broken since 3.4.0
Type: performance Stage: needs patch
Components: Interpreter Core Versions: Python 3.5
Status: open Resolution: not a bug
Dependencies: Superseder:
Assigned To: pitrou Nosy List: Alexey Kazantsev, Vadim Markovtsev, amaury.forgeotdarc, martin.panter, pitrou, rhettinger, serhiy.storchaka, tim.peters
Priority: low Keywords:

Created on 2015-03-20 12:56 by Alexey Kazantsev, last changed 2015-03-28 10:07 by terry.reedy.

File name Uploaded Description Edit Alexey Kazantsev, 2015-03-20 12:56 Alexey Kazantsev, 2015-03-21 11:06
Messages (13)
msg238661 - (view) Author: Alexey Kazantsev (Alexey Kazantsev) Date: 2015-03-20 12:56
Pythons prior to 3.4.0 print


while >=3.4.0 print


If we replace Main with Vector on line 21, the behavior becomes random: in 50% of all cases it prints the wrong sequence, in other 50% the right. Our team treats this as a bug for several reasons:

1) Objects should be destroyed in breadth first reference tree traversal order, starting from the root. There are no cycles. It is nonsense to have freed children in parent's destructor.

2) Our applications suffer very much from this bug. Real "Vector" holds GPGPU memory and real "Device" holds the context, and CUDA/OpenCL require the context to be freed the last. With CUDA, the invalid destructor call order leads to segmentation faults.

This may have something to deal with the implementation of PEP 442 (though in our case there no reference cycles at all).
msg238662 - (view) Author: Vadim Markovtsev (Vadim Markovtsev) Date: 2015-03-20 12:58
msg238669 - (view) Author: Amaury Forgeot d'Arc (amaury.forgeotdarc) * (Python committer) Date: 2015-03-20 13:33
Actually there *is* a cycle:
  assert a.vector is a.vector.device.__class__.__del__.__globals__['a'].vector

A workaround is to not store objects with __del__ in module globals...
Or have only one (the Main instance in your case)
msg238680 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-20 14:30
Amaury is right. In your case you could keep track of the Vectors in the Device object and invalidate them when the Device is destroyed (using e.g. a WeakSet). Or Vector could delegate its destruction to Device, e.g.:

class Device(object):        
    destroyed = False

    def __del__(self):
        self.destroyed = True

    def _dealloc_vector(self, v):
        if not self.destroyed:

class Vector(object):        
    def __init__(self, device):
        self.device = device
    def __del__(self):
msg238787 - (view) Author: Alexey Kazantsev (Alexey Kazantsev) Date: 2015-03-21 11:06
Ok, even assuming that all module globals are in circular reference starting with python 3.4, here is another example without using the globals:

Brief description:
v holds reference to d
a.v = v
b.d = d
Now when we form a circular reference a <-> b, the destructor order becomes wrong for v and d.
msg238788 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-21 11:21
There is a cycle for every class with a method.

>>> class A:
...     def __del__(self): pass
>>> A.__del__.__globals__['A']
<class '__main__.A'>
msg238790 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-21 11:38
There is a cycle involving the class object, but I don’t think there is a cycle involving the instance objects this time.

However, I wonder if __del__() is meant to be called in any particular order anyway. What’s to stop the garbage collector itself from creating a temporary reference to the Device instance, destroying the Vector instance, which invokes Vector.__del__(), and finally destroying the temporary reference to the Device instance?
msg238791 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2015-03-21 11:41
Alexey, you're right that in this case ( the cyclic GC is a bit less friendly than it was. It's not obvious there's a way to change that without introduce a lot of complexity. I'll try to take a look some day, although others may help too :-)

That said, my advice in msg238680 still holds. When you're writing a Python wrapper around a C or C++ library with well-defined ownership relationships, I think you should enforce those in the Python wrapper as well (that's my experience with llvmlite, anyway).
msg238792 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-21 11:50
> There is a cycle involving the class object, but I don’t think there is a cycle involving the instance objects this time.

It is.

>>> a = A()
>>> a.__class__.__del__.__globals__['a']
<__main__.A object at 0xb702230c>

And all objects referenced from user class or module level instance of user class are in a cycle. This problem can cause issues such as issue17852.
msg238794 - (view) Author: Martin Panter (martin.panter) * (Python committer) Date: 2015-03-21 11:55
But in the case of, “a” is a variable inside main(), not a global variable.

BTW I take back my second paragraph questioning the whole order thing; I clearly didn’t think that one through.
msg238796 - (view) Author: Serhiy Storchaka (serhiy.storchaka) * (Python committer) Date: 2015-03-21 12:39
Yes, in we have different cycle.

a ↔ b
↓   ↓
v → d

a and b are in a cycle, and therefore v and d are in cycle. I think that in such case v always should be destroyed before d, independently of a cycle that refers them. And this is the same situation, as for io classes. A TextIOWrapper object refers a BufferedWriter object, a BufferedWriter object refers a FileIO object. and some cycle refers a TextIOWrapper object. As a result a FileIO object can be closed before a TextIOWrapper object or a BufferedWriter object flush its buffer.
msg238829 - (view) Author: Raymond Hettinger (rhettinger) * (Python committer) Date: 2015-03-21 19:40
Can this be closed as not-a-bug.
msg239437 - (view) Author: Terry J. Reedy (terry.reedy) * (Python committer) Date: 2015-03-27 21:23
While this is not a bug, Antoine said he might look at improving the situation, so I leave it to him to close it or not.
Date User Action Args
2015-03-28 10:07:58terry.reedysetnosy: - terry.reedy
2015-03-27 22:44:11rhettingersetpriority: normal -> low
assignee: pitrou
resolution: not a bug
2015-03-27 21:23:16terry.reedysetversions: - Python 3.4
nosy: + terry.reedy

messages: + msg239437

type: behavior -> performance
stage: needs patch
2015-03-21 19:40:36rhettingersetnosy: + rhettinger
messages: + msg238829
2015-03-21 12:39:25serhiy.storchakasetmessages: + msg238796
2015-03-21 11:55:39martin.pantersetmessages: + msg238794
2015-03-21 11:50:34serhiy.storchakasetmessages: + msg238792
2015-03-21 11:41:23pitrousetnosy: + tim.peters
2015-03-21 11:41:09pitrousetmessages: + msg238791
2015-03-21 11:38:04martin.pantersetnosy: + martin.panter
messages: + msg238790
2015-03-21 11:21:17serhiy.storchakasetnosy: + serhiy.storchaka
messages: + msg238788
2015-03-21 11:06:08Alexey Kazantsevsetstatus: closed -> open
files: +
resolution: rejected -> (no value)
messages: + msg238787
2015-03-20 14:30:29pitrousetstatus: open -> closed
resolution: rejected
messages: + msg238680
2015-03-20 14:15:45amaury.forgeotdarcsetnosy: + pitrou
2015-03-20 13:33:34amaury.forgeotdarcsetnosy: + amaury.forgeotdarc
messages: + msg238669
2015-03-20 12:58:55Vadim Markovtsevsetnosy: + Vadim Markovtsev
messages: + msg238662
2015-03-20 12:56:48Alexey Kazantsevsetcomponents: + Interpreter Core
2015-03-20 12:56:26Alexey Kazantsevcreate