Python C API - PyDict - Do values and keys need to be reference counted?

Question

I have some code that is using PyDict internally to store data. A user sets the data like you would with a normal python dictionary, and the data is stored by the C code in the PyDict object. My question is this: Do I need to reference count (Py_INCREF) the value AND the key? Just the value? The documentation indicates that calling PyDict_SetItem does NOT steal a reference for the value. My understanding is that this means the PyDict object is not responsible for the value PyObject, so the reference count needs to be incremented. No mention is made of the key in the docs - does it need to be incremented as well?

A related question involves the deallocator for my code - these reference counts need to be cleaned up - is there any "short cut" to iterate on iterables in the C API and decrement all references, or will I need to manually iterate on the PyDict object and decrement all the values individually (and possibly the keys, depending on the response to the above?)

The code in question would look something like this (with questions in the comments):

int myobject_setitem(myobject *self, PyObject *key, PyObject *value) {

    if (value) {
        Py_INCREF(key); // <--- is this needed?
        Py_INCREF(value);
        return PyDict_SetItem(self->data, key, value);
    }
    /* I assume I need to look up the value so I can decrement the reference count
       before removing it from the dictionary, right? */
    PyObject *value = PyDict_GetItem(self->data, key);
    if (!value) {
        return -1;
    }

    int ret = PyDict_DelItem(self->data, key);
    Py_DECREF(key); // <------- is this needed?
    Py_DECREF(value);
    return ret;
}

As far as the destructor/deallocator, I assume I'd just have to iterate on the dict manually and decrement the values (and maybe keys?) unless there is a better way? (To be clear I mean in the tp_dealloc function)

user2357112 · Accepted Answer · 2020-12-23T01:02:33.207

1

You do not need to perform any reference count handling here. The dict owns all references it holds, and it will manage the incref/decref calls. You do not own any of the references being created or destroyed. You also do not need to call PyDict_GetItem.

The same goes for your deallocator. You only own the reference to the dict itself, not the dict's references to its contents. You only need to decref the dict. The dict will handle its contents.

edited Dec 23 '20 at 01:02

answered Dec 23 '20 at 00:56

user2357112

260,549
28
431
505

So have I completely misunderstood the terminology? The docs indicate the reference is not stolen, doesnt this mean there is no new reference? If the object in question goes out of scope with the caller it won't be garbage collected out from under the dictionary? – Bryant Dec 23 '20 at 01:00
1

@Bryant: You've gotten things backward. If PyDict_SetItem did steal references, *then* you would need to perform incref calls (because you don't own the `key` or `value` references - those are borrowed - so you would need to create new references for it to steal). – user2357112 Dec 23 '20 at 01:05
You are indeed correct, a more careful reading of the docs was in order - thanks! – Bryant Dec 23 '20 at 01:08

Python C API - PyDict - Do values and keys need to be reference counted?

1 Answers1