Python application leaking memory, but Valgrind says no

Question

I have a long-running Python app which scans many megabytes of files in a loop every few minutes. Over the course of a day I see that it gobbles up gigybates of memory, and in the end I have to kill and re-start it. Of course I'm suspecting a C extension I have built myself.

When I run the app under valgrind, though, it shows no leaks (except for some big number in "possibly lost", but the "naked" Python interpreter does that as well).

I've downloaded and compiled a debugging-enabled version of Python-3.6.8 and installed it in a virtualenv but I can't for the life of me figure out how to run setup.py on my extension with debugging enabled, let alone how to install all the other packages I need to run the full application.

Answer 1

It turns out that over-referenced PyObjects end up in Valgrind's "Possibly lost" category, not "definitely lost". But even when running the Python interpreter alone there is a lot of stuff in "possibly lost," so it took me a while to figure out where my code leaked memory (during the course of development a call to PyList_SetItem() had transmogrified into PyList_Append(), which doesn't steal the item's reference).