This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: refcounts not respected at process exit
Type: Stage:
Components: Interpreter Core Versions: Python 3.4
process
Status: closed Resolution: not a bug
Dependencies: Superseder:
Assigned To: Nosy List: minrk, nathan.stocks, pitrou, tim.peters, vstinner
Priority: normal Keywords:

Created on 2014-04-25 19:39 by minrk, last changed 2022-04-11 14:58 by admin. This issue is now closed.

Files
File name Uploaded Description Edit
tstgc.py minrk, 2014-04-25 19:39 failing test case
Messages (11)
msg217168 - (view) Author: Min RK (minrk) * Date: 2014-04-25 19:39
Reference counts appear to be ignored at process cleanup, which allows inter-dependent `__del__` methods to hang on exit. The problem does not seem to occur for garbage collection of any other context (functions, etc.).

I have a case where one object must be cleaned up after some descendent objects. Those descendents hold a reference on the parent and not vice versa, which should guarantee that they are cleaned up before the parent. This guarantee is satisfied by Python 3.3 and below, but not 3.4.

The attached test script hangs at exit on most (not all) runs on 3.4, but exits cleanly on earlier versions.
msg217169 - (view) Author: Nathan Stocks (nathan.stocks) Date: 2014-04-25 19:49
This affects me as well.  I have to manually clean up objects in the correct order in script I am working on under 3.4.0.  I have this problem under both OS X 10.9.2 Mavericks and under CentOS 6.5
msg217172 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2014-04-25 21:19
Just noting that, for me, the problem goes away if

del c, c2

is added as the last line of the test.  This suggests the problem is due to changes in end-of-life module cleanup.

Without that line, I see 3 kinds of output:

1.
del child
del child
del parent
parent deleted

2.
del parent
parent still has 2 children
parent still has 2 children
... repeated forever ...

3.
del child
del parent
parent still has 1 children
parent still has 1 children
... repeated forever ...
msg217177 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-26 00:43
(the 3 kinds of output are probably due to hash randomization)
msg217178 - (view) Author: Antoine Pitrou (pitrou) * (Python committer) Date: 2014-04-26 01:00
Looking into it, it's normal for refcounts to be "ignored": those objects belong to reference cycles:

tstgc.__dict__
-> p (or c, or c2)
-> p.__class__ (i.e. Parent, or Child respectivel))
-> Parent.__dict__
-> Parent.__del__ (or Parent.__init__, or Parent.child)
-> Parent.__del__.__globals__ (which is tstgc.__dict__)

Since p, c, c2 belong to reference cycles, they get collected in an undefined order.

Obviously, Parent.__del__ is buggy (it runs into an infinite loop when self.children != 0).

Before Python 3.4, the module globals would have been set to None at shutdown, which would have broken those cycles, but caused other well-known problems. It's probably impossible to find a scheme that satisfies all constraints, so we'll see in the future if the new scheme brings more drawbacks than advantages (right now, my own evaluation is obviously that it's a step forward).
msg217179 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2014-04-26 01:35
I think Antoine is right on all counts.  The most surprising bit may be that p, c, and c2 are in reference cycles, but - surprising or not - that's always been true.  The reason "it worked" before 3.4 is that CPython happened to break the cycles via the nasty hack of binding each module global to None at shutdown.

minrk, note that gc in CPython does not (for example) run in a separate thread.  That's why, when it triggers, the infinite loop in your Parent.__del__ will in fact run forever.  gc runs in the same thread (the main thread) as Parent.__del__, so spinning in the Parent.__del__ loop prevents anything else (including more gc) from ever being done.

Take out the infinite loop, and all three objects (p, c, c2) are collected.  But the order in which they're collected isn't defined (because they're all in cyclic trash), and even changes from run to run because hash randomization changes the order in which they appear when traversing testgc.__dict__.

An interesting question remaining is how you _could_ force a finalization order in this case, in a way that doesn't rely on implementation accidents.  A clean way doesn't spring to my mind immediately.
msg217236 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2014-04-27 01:01
After more thought, I don't think the user can do anything to influence finalization order in cases like this, short of adding "del" statements (or moral equivalents) to break cycles before the interpreter shuts down.

Fine by me ;-)  Something CPython could do, when collecting cyclic trash, is pick on objects with the smallest refcount first.  That would most often mimic the finalization-order effects of the old bind-module-globals-to-None hack.  But it would require more code and more expense to impose a partial order, and would still be an implementation detail specific to CPython (the implementation, as opposed to Python the language).
msg217666 - (view) Author: Min RK (minrk) * Date: 2014-04-30 23:57
Thanks for clarifying that there is indeed a reference cycle by way of the module, I hadn't realized that.

The gc blocking behavior is exactly why I brought up the issue. The real code where this causes a problem (rather than the toy example I attached) is in pyzmq, where destroying a Context object calls `zmq_term`, a GIL-less C call that will (and should) block until all associated sockets are closed. Deleting a socket closes it. Sockets hold a reference to the Context and not vice versa, which has ensured that the sockets are collected before the Context until Python 3.4. 

Does this mean it is no longer possible to express that one object should be cleaned up before another via references?

I think I will switch to adding an atexit call to set a flag that prevents any cleanup logic during the atexit process, since it does not appear to be possible to ensure deletion of one object before another in 3.4.
msg217667 - (view) Author: Tim Peters (tim.peters) * (Python committer) Date: 2014-05-01 00:17
There's no way to influence finalization order for objects in cycles, and there never was.  So nothing actually changed in that respect ;-)  What did change is that Python used to forcibly break many module-level cycles in a way that just happened to result in the finalization order you wanted.

Outside of cycles it's still as predictable as before.  For example, you could reliably get the finalization order you want by replacing the last two lines with

p.child()
p.child()

Then the children aren't in cycles, and will be finalized first.  But that's still due to other CPython implementation details (which haven't yet changed), not to language guarantees.

But an atexit handler sounds like a saner way to proceed :-)
msg217672 - (view) Author: Min RK (minrk) * Date: 2014-05-01 00:42
Thanks for your help and patience. Closing as slightly unfortunate, but not unintended behavior.
msg217682 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2014-05-01 07:39
Never rely on the GC. Avoid cycles by using the weakref module.
History
Date User Action Args
2022-04-11 14:58:02adminsetgithub: 65550
2014-05-01 07:39:07vstinnersetmessages: + msg217682
2014-05-01 00:42:18minrksetstatus: open -> closed
resolution: not a bug
messages: + msg217672
2014-05-01 00:17:29tim.peterssetmessages: + msg217667
2014-04-30 23:57:10minrksetmessages: + msg217666
2014-04-27 01:01:14tim.peterssetmessages: + msg217236
2014-04-27 00:31:02vstinnersetnosy: + vstinner
2014-04-26 01:35:27tim.peterssetmessages: + msg217179
2014-04-26 01:00:41pitrousetmessages: + msg217178
2014-04-26 00:43:59pitrousetmessages: + msg217177
2014-04-25 22:07:49tim.peterssetnosy: + pitrou
2014-04-25 21:19:03tim.peterssetnosy: + tim.peters
messages: + msg217172
2014-04-25 19:49:10nathan.stockssetnosy: + nathan.stocks
messages: + msg217169
2014-04-25 19:39:06minrkcreate