Creating a class (either using the 'class' statement or using type()) creates a circular reference.

I've attached a script that simply demonstrates this. The problem is caused by the fact that MyClass.__dict__['__dict__'].__objclass__ is MyClass.

Although most of the times classes are never deleted when the interpreted exits, some programs (like the popular Django framework) create temporary classes. And this is a pain because you can't disable the garbage collector.

Is there a reason why you want to disable the cyclic garbage collector?

This should not be a problem, I think. We cannot avoid such a circular reference. 

I believe the main reason of the circular reference is in MyClass.__mro__, this is a list and contains a reference to MyClass. You can add "__slots__ = []" to MyClass's class body to avoid __dict__(the one you mentioned) and another member which contains two more references to MyClass, but the __mro__ still contains the last reference to MyClass.

Disabling the GC can increase performances (although not significantly). But this bug is the cause of other problems too: what if the metaclass contains a __del__() method?

An another issue that I've found is that debugging is harder. I always try to avoid to create ref cycles in my code, also if my objects are collectable. In this way, I'm sure that I'll always be able to add a __del__ method in the future without problems. However, I can't easily check ref cycles without manually inspecting `gc.garbage`.

And also, specifying DEBUG_SAVEALL will put all the deleted classes and their attributes in the garbage, which makes debugging *very* hard in case of a leaking program.

I couldn't imagine why a metaclass would want to have a __del__ method...

Having a __del__ inside a metaclass is strange, I know... but probably there are situations where you need to do so. Why shouldn't a developer be able to add a __del__ to a metaclass without creating uncollectable objects? I don't think this behavior is by design.

Also, doing random contributions to various projects I've seen many odd things. However I don't think that the bug tracker is the right place to discuss development practices. A bug that causes problems in unusual situations is still a bug. :-)

Whether this is a bug is not clear.  Sometimes it's just impossible not to create cycles, and classes may just be one instance of that.

This is an unwanted an unexpected behavior, so this is a bug by definition. If it's not easy to fix, it's a different matter.

However here's a proposed solution:

* for the __mro__: instead of using a tuple, use a new object that inherits from it. This new object should use weak reference for the first item and should return the real object (if available) only in __getitem__().

* __objclass__ can should become a property based on weak references.

This was discussed in issue1545463 which was deemed not to be worth fixing.  The particular bug described in issue1545463 would also be fixed by the still open issue812369.  So yes, there are cases where these cycles are a problem even with gc.

This is not so easy: the __mro__ tuple, as its name says, is used internally for method resolution, or finding attributes on the type's bases. __objclass__ is used whenever the descriptor is accessed.  These operations are involved in every method call.

If you want this to go somewhere, you will have to produce a patch and demonstrate that the slowdown does not unsuitably impact performance.

Interesting.  Here is a case for disabling GC:  

We intentionally disable GC in Eve-online, because of the unpredictable lag spikes it produces.  In a server application with gigabytes of python objects, GC can cause hickups of some seconds, which is unacceptable when trying to keep the latency low.

We were considering using Django for our backend web framework once.  Good that we didn't, since it would have leaked.

We do try to keep our code free from circular references, and occasionally run single GC passes on our test cluster to weed out any that may have formed accidentally (gc.garbage with DEBUG_LEAKS)

IMHO, python should try to be as free from these as possible, although it is admittedly not always easy (see recursive closures, issue 7464)

An alternative, is to do something like the minidom module does:  Provide a "unlink" method (or similar) to manually nerf objects before forgetting them.  Perhaps this could be standardized with an unlink keyword and a __unlink__ special method?

> However here's a proposed solution:

* for the __mro__: instead of using a tuple, use a new object that inherits from it. This new object should use weak reference for the first item and should return the real object (if available) only in __getitem__().

* __objclass__ can should become a property based on weak references.



I'm afraid doing so could cause some public C API change. For example, the PyDescr_TYPE, if we implemented all the Descriptor object's d_type(that is the __objclass__ of all types of descriptors) based on weakref, we could have all the callers who call the descriptor functions to check weather the weak-referented class object has gone away.

I can confirm this bug should be addressed some way. On my high-traffic server, keeping GC enabled causes performance issues due to this bug, and disabling it causes an out-of-memory within hours.

Is this due to your adopting of Django?
Are there other guys who using Django suffer the same problem?

Andrea: for the purpose of this tracker, a 'bug' is a discrepancy between doc and behavior. The fact that (new-style) classes have circular references was known when they were introduced in 2.2. That fact is not a bug in the above sense. It is a design tradeoff that works OK for the typical case of classes remaining until the end of a program run. Changing the tradeoff embodied in type() is not likely to be accepted.

That said, an alternate no_cr (no circular reference) metaclass (perhaps a type subclass) that contructed cr-free classes, even at the cost of slower lookup, might be a good addition to the stdlib. Ray's ideas seems to me at least plausible. My first thought on location is in functools.

What about my suggestion of simply providing a convention on how to disable links manually, similar to how minidom.document does it with an "unlink" method?  If these cases are documented, then there shouldn't be any extra bother to do so.
We could provide an builtin unlink() function that calls any __unlink__ special method if present.

This sounds like a feature request, really, and one that is unlikely to get accepted since it sort of exposes an implementation detail of C python.  Oh well.

krisvale:

> What about my suggestion of simply providing a convention on how to disable links manually, similar to how minidom.document does it with an "unlink" method?  If these cases are documented, then there shouldn't be any extra bother to do so.
We could provide an builtin unlink() function that calls any __unlink__ special method if present.


Maybe through this way such problems can be solved, but it seems it's not graceful and pythonic enough. The reason such an "unlink" method appears in xml.dom.minidom is because usually large xml document can eat up so much memory that we had to do something to make the memory garbaged sooner. But adding this as a standard __unlink__ method and requiring explicitly call the unlink() method make it similar as using c & c++, which require explicitly memory release. 

Besides, as you mentioned, "exposes an implementation detail of C python", it's better that the object reference mechanism be fully controlled by python core and not be awared by users at all. That is also one of the purposes of designing python. 

I prefer terry's idea, which suggests adding a metaclass that produces rc-free class to stdlib. These class should behave almost the same as the ones "type" metaclass produces. In order not to introduce the slower lookup, we can also add a little differences on usage of such classes, for example, internally store mro list without the class itself, but construct a full mro list when getting the __mro__ attribute, and different "super" implementation, since explicitly getting the mro is not so common in practice.

I am opposed to a new 'unlink' builtin and '__unlink__' special method. There are more than enough builtin functions now, and this need is specific to those who both make lots temporary classes that they need garbage collected AND need to run with gc turned off. The intersection of these two small groups is probably very small. Since turning gc off is only an option with CPython (and PyPy?), this need is also implementation specific.

Before anything were added to the stdlib (and the need *might* be considered too rare for that) it should be tested in real use by more than one person. It appears that there are at least a couple of options for implementing a no_cr class. They could be attached to this issue, posted on the cookbook, or uploaded to PyPI.

Does anyone wish to take this forward, is it simply dead in the water or what?  You might like to note that msg111963 refers to #812369 which was superseded by #18214 which was fixed in 3.4.

I don't think this can go somewhere useful.

Today I changed my reference count tests to not use debug Python and came across this issue.

From my testing, Python3.4 is the first affected version, Python3.3 was still fine, so were 2.7 and 2.6. 

This leaks:

def simpleFunction39():
    class Parent(str):
        pass

This does not:

def simpleFunction39():
    class Parent(object):
        pass

When comparing (or attempted to) gc.get_objects() before and after, I was unable to point to any count that would be different, that was confusing. I can rule out a mistake in my changes to how the counts are achieved, because every other reference count test works.

What I also meant to say, is that debug Python is not affected, and this had me deeply confused. Any ideas how that could happen?

Kay, please ask your question on python-list or a similar forum.