setattr, object deletion and cyclic garbage collection

Question

I would like to understand how object deletion works on python. Here is a very simple bunch of code.

class A(object):

    def __init__(self):
        setattr(self, "test", self._test)

    def _test(self):
        print "Hello, World!"

    def __del__(self):
        print "I'm dying!"

class B(object):

    def test(self):
        print "Hello, World!"

    def __del__(self):
        print "I'm dying"

print "----------Test on A"
A().test()
print "----------Test on B"
B().test()

Pythonista would recognize that I'm running a python 2.x version. More specially, this code runs on a python 2.7.1 setup.

This code outputs the following:

----------Test on A
Hello, World!
----------Test on B
Hello, World!
I'm dying

Surprisingly, A object is not deleted. I can understand why, since the setattr statement in __init__ produces a circular reference. But this one seems to be easy to resolve.

Finally, this page, in python documentation (Supporting Cyclic Garbage Collection), show that it's possible to deal with this kind of circular reference.

I would like to know:

• why I never go thru my __del__ method in A class?
• if my diagnosis about circular reference is good, why my object subclass does not support cyclic garbage collection?
• finally, how to deal with this kind of setattr if I really want to go thru __del__?

Note: In A if the setattr points to another method of my module, there's no problem.

Answer 1

Fact 1

Instance methods are normally stored on the class. The interpreter first looks them up in the instance __dict__, which fails, and then looks on the class, which succeeds.

When you dynamically set the instance method of A in __init__, you create a reference to it in the instance dictionary. This reference is circular, so the refcount will never go to zero and the reference counter will not clean A up.

>>> class A(object):
...     def _test(self): pass
...     def __init__(self):
...             self.test = self._test
... 
>>> a = A()
>>> a.__dict__['test'].im_self

Fact 2

The garbage collector is what Python uses to deal with circular references. Unfortunately, it can't handle objects with __del__ methods, since in general it can't determine a safe order to call them. Instead, it just puts all such objects in gc.garbage. You can then go look there to break cycles, so they can be freed. From the docs

gc.garbage

A list of objects which the collector found to be unreachable but could not be freed (uncollectable objects). By default, this list contains only objects with __del__() methods. Objects that have __del__() methods and are part of a reference cycle cause the entire reference cycle to be uncollectable, including objects not necessarily in the cycle but reachable only from it. Python doesn’t collect such cycles automatically because, in general, it isn’t possible for Python to guess a safe order in which to run the __del__() methods. If you know a safe order, you can force the issue by examining the garbage list, and explicitly breaking cycles due to your objects within the list. Note that these objects are kept alive even so by virtue of being in the garbage list, so they should be removed from garbage too. For example, after breaking cycles, do del gc.garbage[:] to empty the list. It’s generally better to avoid the issue by not creating cycles containing objects with __del__() methods, and garbage can be examined in that case to verify that no such cycles are being created.

Therefore

Don't make cyclic references on objects with __del__ methods if you want them to be garbage collected.

Answer 2

You should read the documentation on the __del__ method rather carefully - specifically, the part where objects with __del__ methods change the way the collector works.

The gc module provides some hooks where you can clean this up yourself.

I suspect that simply not having a __del__ method here would result in your object being properly cleaned up. You can verify this by looking through gc.garbage and seeing if your instance of A is present.