How to reliable list code paths that keeps a variable from being garbage collected in python?

Question

I'm trying to diagnose memory leaks caused by Exceptions.traceback and I would like to be able to list all or most of the paths that leads to the variable that should be garbage collected but isn't.

I'm currently using a bit clumsy code to print out the reference graph, but I hoped there is a library or tool that have this ability build in. Ideally with some nice way to dump the graph and then explore it later interactively.

You can see my current approach (functions print_ref_graph and find_tracebacks), here: https://nbviewer.org/gist/PiotrCzapla/1ff0fa083e8a4ca657ad86b1942abf42

Answer 1

While looking for memory profilers I've found some tools that are able to show the dependency graph. The best so far is objgraph. For an unhandled exception it is able to nicely show where the object lives.

objgraph

It works much faster than pympler and show nice visualisation in jupyter notebook. So I really recommend this one.

To find all places that holds your last unhandled exception objgraph.show_backref(sys.last_traceback) is what you need.

If you want to find out if you have some objects of particular type that are still alive, it has a method for that: objgraph.by_type('type name') to return a list.

pympler

Its interactive reference browser does not work on macOS, but file browser gave me an output that is super slow but acceptable. (the only bit missing is that it does not list keys where the object is referenced).

The following code listed most of the places where sys.last_traceback is referenced in jupyter. But without keys in dict you won't know that sys holds traceback in last_traceback, and AutoFormattedTB has tb attribute.

from pympler import refbrowser
ib = refbrowser.FileBrowser(sys.last_traceback)
ib.print_tree('out.txt')