1

I'm trying to pass and access a Numpy Structured Array created in Python from the C part of an application. The usage I want to make of such array is to create the array from the Python side, use it as I will (still from Python) and then for heavy-computation phases pass the array to a C module (no copies, using pointers) so that data can be accessed and modified in the C side.

From the Python-side, I'm creating a Numpy structured array with a custom dtype such as the following, and passing it to a function located in a simple C extension:

Python

import numpy
import cmodule

blocksize=10
numblocks=5

particle_struct_dtype = numpy.dtype([
    ('position_x', 'f4', (blocksize,)),
    ('position_y', 'f4', (blocksize,)),
    ('position_z', 'f4', (blocksize,))
])

particles = numpy.empty(numblocks, dtype=particle_struct_dtype)

cmodule.compute(particles, blocksize, numblocks)

As shown, I'm creating an array of key-values, where each key represents the "field" of a C struct, and each value of such field is an array of floats. Then, in the C side, I try to obtain the array by:
- First using the PyArg_ParseTuple function to retreive a reference to the array
- Then using a dummy PyObject created by Py_BuildValue to obtain a representation of the dtype of the array (the structure's type), to later cast it with PyArray_AsCArray

C

static PyObject *compute(PyObject *self, PyObject *args)
{
    PyArrayObject *particles;
    int numblocks, blocksize;

    // Parse arguments
    if (!PyArg_ParseTuple(args, "O!ii", &PyArray_Type, &particles, &blocksize, &numblocks)) {
        Py_RETURN_NONE;
    }
    PyArray_Descr* arrayDescr;
    PyObject* dummy;

    dummy = Py_BuildValue("[(s, O), (s, O), (s, O)]", "position_x", (blocksize), "position_y", (blocksize), "position_z", (blocksize));
    PyArray_DescrConverter(dummy, &arrayDescr);

    struct particles_t *castedParticles;
    npy_intp dims[1];
    dims[0] = blocksize;
    PyArray_AsCArray((PyObject **)&particles, (struct particles_t**)&castedParticles, dims, 1, arrayDescr);
    ...
}

Nonetheless, I keep getting SEGFAULT errors when accessing the data. Digging into the errors, I found that the data is not casted as I thought it would be (the conversion is not properly done).
I have tried to simplify the example by passing an array of simple tuples instead of an array of tuples with more arrays inside them:

dummy = Py_BuildValue("[(s, s), (s, s), (s, s)]", "position_x", "f4", "position_y", "f4", "position_z", "f4");
PyArray_DescrConverter(dummy, &arrayDescr);

struct simple_particles_t *castedParticles;
npy_intp dims[1];
dims[0] = blocksize;
PyArray_AsCArray((PyObject **)&particles, (struct simple_particles_t**)&castedParticles, dims, 1, arrayDescr);

And this seems to work, I obtain the array of simplified structures, each only holding 3 floats (one per field) instead of 3 arrays of floats.

My doubts are:
- Am I missing something? Is it even possible to cast the structured numpy array, where each field has an array of floats, to a C array of structures, so that each Python-field is mapped to the C structure's field? As an example, accessing particles[0].position_x[0] from C would be like accessing particles[0]['position_x'][0] from the Python side.
- Is this going to create any copies? I am trying to access and modify the memory that is allocated by Python from C.

Pyramid
  • 31
  • 8
  • I haven't tried to understand your C structure, but can comment on the `dtype`. One record of the array will have 30 floats. 10 are accessed as 'position_x', next 10 as 'y' etc. x,y,z values within a record are not interleaved. – hpaulj Jun 11 '19 at 16:17
  • Hey @hpaulj, thanks for the reply. What you mention is the behavior I expect. I want ```position_x``` to be an array of 10 (just an example) floats together in memory, then ```position_y``` the next 10, etc. The problem is, the ```[(s, O), ...]``` part is probably not right, as the "O" from PyObject does not seem to translate to an array of floats. – Pyramid Jun 11 '19 at 16:23

0 Answers0