2

So I have put together a few Python C-extensions and although their respective behaviors are verified, I'd like to verify this by some refcount debugging.

How can I verify that I have inserted the INC/DEC refcounts properly? I want to do this verification from a black box perspective, that is, how should a function's input/output/memory look if all refcounts have been done properly?

My first approach would be to use sys.getrefcount(obj) and check all the input/output objects' refcounts to see they are ok. Secondly, I could also perhaps check the memory used to see that there is no memory leak.

But I have never really done this before, so what's the easiest and most correct way to do this?

Please note: I don't want to use any Cython style libraries, I'd rather do the C nitty gritty myself to learn the fundamentals.

Solution: So as suggested I built a debug enabled Python interpreter. I thought it would be tricky, but it wasn't. As per this recipe I added those exact lines to Objects/object.c. After that all I had to do in the downloaded Python source directory was:

./configure --with-pydebug --prefix=/usr/local/python/
make
make install

With regards to building the C-extension to this new interpreter, all that had to be done on that front was to point the include_dirs tag in the Extension in setup.py to '/usr/local/python/include/python2.7', and then running the build and install command with the new interpreter. Once that was done I could add just add _Py_CountReferences(stderr); lines in the c code and the refcount would be dumped to stderr (console). Very good to see what changes the refcount, and how it increments/decrements as the code progresses.

c00kiemonster
  • 22,241
  • 34
  • 95
  • 133

1 Answers1

1

One suggestion: Compile the Python interpreter in debug mode (--with-pydebug). Use this interpreter to compile and test your extension. The interpreter displays the count of the total number of objects and this total shouldn't increase as you repeated run your tests. Make sure to test all the possible exit paths of your code - I've forgotten to DECREF objects when an exception occurred.

casevh
  • 11,093
  • 1
  • 24
  • 35
  • I read about that in some blog post. I thought perhaps I could do it without having to compile the interpreter. But if it tells me everything I need to know I guess that's the best way of doing it. I can set up a virtualenv for this, right? – c00kiemonster Dec 28 '11 at 03:20
  • I don't have any experience with virtualenv. A quick scan of the documentation implies virtualenv creates different environments for the same version of Python. In this case, you'll be compiling a slightly different version of Python. I don't know if virtualenv will help with that or not. – casevh Dec 28 '11 at 03:56
  • Ok so I managed to build a debug version of Python, and my C-extension links and compiles against it. But now what, how do I access any of the information? I'm trying to get _Py_CountReferences to work from [this](http://code.activestate.com/recipes/67112-help-in-debugging-memory-problems/) example (taken from the Python cookbook), but it won't work. The compiler complains when I try to insert a `COUNTREFS();` line. – c00kiemonster Dec 28 '11 at 05:50
  • Based on a quick read through the link, it looks like it wants you to modify the source of Python itself and then define macros that create COUNTREFS(). I haven't tried that. I normally verify the refcount of all the returned objects is correct and that the total refcount doesn't increase as I repeat my tests. The total refcount can be access (from a debug build) as sys.gettotalrefcount(). – casevh Dec 28 '11 at 06:08
  • Yea I skipped that old recepie, and just used gettotalrefcount() instead. Where can I find info on other sys methods with Python debug mode? Thanks much! – c00kiemonster Dec 28 '11 at 06:23
  • I think the only other one is getobjects() but I've never used it. – casevh Dec 28 '11 at 06:48