1

I am trying to wrap a concurrent queue implementation in C with Python. It has two methods:

  • push adds a generic void pointer (in this case a PyObject pointer) to an internal linked list and increments its reference count by one.
  • pop removes an object pointer from the linked list and returns it.

In the internal linked list each node holds a pointer to the stored object, and a pointer to the next node. Each operation activates one of two mutexes to prevent race conditions (if you are curious, it is a Michael-Scott two-lock concurrent queue). As I mentioned, I want to use this queue to store Python objects.

Reading the official Python documentation I came upon cyclic garbage collection, which is enabled with the Py_TPFLAGS_HAVE_GC flag. From what I understand, any Python type that is a container for other types (including other containers as well) must support it, and my queue seems to fall into this category. Any type that supports this functionality must define the tp_traverse and tp_clear fields in their type struct with their own implementations. For a normal container type (like a list), the tp_traverse function would loop over all the contained elements and call Py_VISIT on them. tp_clear is similar, but it calls Py_XDECREF.

My questions:

  • Should I lock the whole list before looping through all of its elements to perform de adequate calls? Does this lock impact performance or are threads suspended anyway when garbage collection is performed?
  • Should I remove an element from the queue completely after calling Py_XDECREF? My guess is that I should check if the ob_refcnt field is zero for each PyObject in my queue, is this correct?
00xc
  • 21
  • 2
  • 6

0 Answers0