Return a C String Array from a C Extension

Question

I have a C function that returns an array of strings. How can I call it in the form of a Python C extension which will return the array back to a calling Python function? (I am new to Python C extensions and have minimal experience with the extensions)

This is the definition I tried:

static PyObject* _get_array(PyObject* self, PyObject* args)
{
    int64_t value;
    int init_level;
    int final_level;

    if(!PyArg_ParseTuple(args, "Lii", &value, &init_level, &final_level))
        return NULL;

    // returning the array as a Python object by o
    return Py_BuildValue("o", _get_array(value, init_level, final_level));
}

and the method def:

static PyMethodDef array_methods[] = {
    { "get_array", _get_array, METH_VARARGS, "Returns a string array"},
    { NULL, NULL, 0, NULL }
};

Update

get_array function:

#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <Python.h>

char **get_array(int64_t value, int init_level, int final_level) {

  int SHIFTS []= {44, 40, 36, 32, 28, 24, 20, 16, 12, 8, 4, 0};

  long count =  1 << (4* (final_level - init_level));
  char** t_array;
  t_array = malloc(sizeof(char*)*count);

  int shift_coff = 11 -(final_level-init_level);
  int64_t base = (value << SHIFTS[shift_coff]);

  for (long i=0; i < count; i++){
    t_array[i] = malloc((4+final_level)*sizeof(char));
    sprintf(t_array[i], "%llX", (base + i));
  }

  return t_array;
}

Where's the Python code showing how you want to call it? Have you been through any tutorials, such as this one: [Python - Extension Programming With C](https://www.tutorialspoint.com/python/python_further_extensions.htm)? — lurker, Apr 24 '19 at 22:23
Yes I did. I have done extensions that return simple values, like ints or floats, but couldn't find a way to return an array — c00der, Apr 25 '19 at 03:53
I think you need to show the C function you want to call (to show how the return type is allocated, etc). At the moment you just have a nonsensical recursive call to `_get_array`. It's basically pretty easy - you create a Python list and iterate through it to fill it with Python strings - but the details do depend on code you don't show — DavidW, Apr 25 '19 at 09:39
@DavidW, sorry for the delay. Just added. My concern is, there can be a lot of items in the array that is returned and don't want to use a Python list because of the overhead. I would much prefer using a pure C array instead. Somewhere I read you can use Numpy arrays but I am confused whether I can return a pure C array or need to use Python or Numpy specfic array implementation — c00der, Apr 25 '19 at 14:49
You definitely can't return a pure C array. Numpy string arrays would work fine, but the strings must be of fixed length (which it looks like these are). Python lists are better than you think. I'll post a proper answer if/when I can (and if no one else does it first) — DavidW, Apr 25 '19 at 15:18
Thanks, since I don't know, why can't a pure C array be returned? Is it because Python can't interpret what that is? Also, when you use a Python list, can C long long still be used or should they be Python_ints? — c00der, Apr 25 '19 at 17:14
Yes - it's that Python doesn't know what it is. If I understand what you want correctly, the Python list would be of strings rather than ints, but you could definitely make the string from a C long long like you do here. — DavidW, Apr 25 '19 at 20:39

score 1 · Accepted Answer · answered Apr 26 '19 at 08:07

You can't return your char** directly Python since Python only understands objects of type PyObject* (since this contains the information needed to handle reference counting and identifying the type). You therefore have to create a suitable Python object. The simplest option would be a list of strings. The next simplest you be a numpy array using the string type (you can do this easily because all your strings are the same length). Neither of these have a direct Py_BuildValue conversion so you have to write loops yourself.

For a list of strings you simply create the list with PyList_New then go through element by element with PyList_SetItem:

char** array = get_array(value, init_level, final_level);
PyObject* list = PyList_New(1 << (4* (final_level - init_level)));
if (!list) return NULL;

for (int i=0; i<(1 << (4* (final_level - init_level))); ++i) {
    PyObject* item = PyBytes_FromStringAndSize(array[i],(4+final_level));
    if (!item) goto failed;

    if (PyList_SetItem(list,i,item) != 0) {
        Py_DECREF(item);
        goto failed;
    }

    free(array[i]); // deallocate array as we go
}
free(array);

// returning the array as a Python object by o
return list;

failed:
Py_DECREF(list);
// also deallocate the rest of array?
return NULL;

Note that I haven't finalised memory management of failure so you'll leak array.

For the numpy array you allocate an array with the correct string type, and then copy the data into it

char** array = get_array(value, init_level, final_level);

// create an "Sx" dtype, where x is a suitable number
PyArray_Descr *desc = PyArray_DescrNewFromType(NPY_STRING);
desc->elsize = (4+final_level);

npy_intp array_length[] = {1 << (4* (final_level - init_level))};
PyObject* nparray = PyArray_SimpleNewFromDescr(1,array_length,desc);
if (!nparray) return NULL; // clean up array too

for (int i=0; i<(1 << (4* (final_level - init_level))); ++i) {
    char* data = PyArray_GETPTR1((PyArrayObject*)nparray,i);

    // copy data
    for (int j=0; j<(4+final_level); ++j) {
        data[j] = array[i][j];
    }

    free(array[i]); // deallocate array as we go
}
free(array);

// returning the array as a Python object by o
return nparray;

Again, not all the error handling is perfect. For this example to work you must call import_array() in your module init function.

In both cases you might be better not allocating memory in get_array but instead writing directly into your Python objects.

Return a C String Array from a C Extension

1 Answers1