C-Extension: Ref count of generators

Question

I'm trying to get the Py_INCREF's and Py_DECREF's for my c extension right. Whilst doing so I've been stumbling about really high values for generators. What I've been doing is the following:

// Forgive me for leaving out the NULL checks
PyObject *get_generator = PyUnicode_InternFromString("get_generator");
PyObject *callback;
PyObject *seq;
PyObject *item;

callback = PyObject_CallMethodObjArgs(parent->state, get_generator, NULL);
seq = PyObject_GetIter(tmp);

Py_DECREF(callback);

while ((item = PyIter_Next(seq))) {
    ...
    Code
    ...
    Py_DECREF(item);
}
Py_DECREF(seq);
printf("!!!%zd\n", seq);

Can somebody explain please.

`seq = PyObject_GetIter(tmp);` - do you mean `seq = PyObject_GetIter(callback);`? — DavidW, Dec 16 '19 at 17:22
And what is `printf("!!!%zd\n", seq);`? Do you think this is printing the reference count? This will print the pointer value. It probably wouldn't be valid to print the reference count there anyway since the object might well have been destructed — DavidW, Dec 16 '19 at 17:25
@DavidW I do mean ```seq = PyObject_GetIter(callback);``` and ```printf("!!!%zd\n", seq);``` is a stupid mistake of mine. I meant to write ```seq->ob_refcnt``` — Nima Mousavi, Dec 16 '19 at 20:00

score 2 · Accepted Answer · answered Dec 16 '19 at 18:45

printf("!!!%zd\n", seq); isn't printing the reference count, it's printing the raw pointer address (slightly incorrectly, since there is no guarantee that a pointer and a signed size_t are compatible for printing).

Also note that it's not safe to check the real reference count if you don't hold a pointer you own (or have borrowed safely), so checking after the Py_DECREF (when you gave up ownership) is undefined behavior.

To fix, change:

Py_DECREF(seq);
printf("!!!%zd\n", seq);

to:

printf("!!!%zd\n", Py_REFCNT(seq));
Py_DECREF(seq);

You should expect to see a count of 1 under normal circumstances, though if the object passed to PyObject_GetIter is itself an iterator (making PyObject_GetIter the identity function, doing nothing but incrementing the reference count and returning the otherwise unmodified argument), then the count could be higher if other references to the same iterator might exist elsewhere.

Assuming they weren't omitted simply for brevity, you do need to add checks for return values; virtually all of your API calls could return NULL if an exception is raised, and silently continuing processing rather than passing the error up immediately risks replacing a (useful) exception message/traceback with an (unhelpful) segfault.

Dooh... sry for my stupid mistake. The refcount is still high but within limits. I don't get why yet, but I might delete this question, as it holds no value for SO. — Nima Mousavi, Dec 16 '19 at 18:52
Even though I forgot the ```seq->ob_refcnt``` and my question was kinda stupid too begin with. Your answer was actually quite helpful. — Nima Mousavi, Dec 16 '19 at 19:58

C-Extension: Ref count of generators

1 Answers1