Nested extension module cannot be imported

Question

I am trying to import an extension module contained inside a nested package. For the purposes of this example, it's the only element of the root module, but the real case has others. When I attempt to test the following module setup using tox (or cibuildwheel/pytest, it's the same result) I get the following error:

ImportError: cannot import name 'dummyext' from 'rootpack'

Here is the code for setup.py:

import setuptools
import numpy

PACKAGE_NAME = 'rootpack'

setuptools.setup(
    name=PACKAGE_NAME,
    version="0.0.1",
    description="Testing extension module",
    packages=[PACKAGE_NAME],
    #ext_package=PACKAGE_NAME,
    ext_modules = [
        setuptools.Extension(
            'rootpack.dummyext',
            ['ext/dummy.c'],
            extra_compile_args=["-std=c99"],
            include_dirs=[numpy.get_include()],
        ),
    ],
    install_requires=[
        'numpy',
    ],
)

And dummy.c:

#include <Python.h>
#include <numpy/arrayobject.h>

static PyObject* dummy_method(PyObject* self, PyObject* args)
{
    npy_int rows = 3;
    npy_int cols = 4;
    npy_intp dims[2] = {rows, cols};
    PyObject* arr_obj = PyArray_ZEROS(2, dims, NPY_INTP, 0);
    int* arr_data = PyArray_DATA(arr_obj);

    for(int i = 0; i < rows; ++i)
    {
        for(int j = 0; j < cols; ++j)
        {
            arr_data[i*cols + j] = i+j;
        }
    }
    return arr_obj;
}

static PyMethodDef dummyext_methods[] = {
    {"dummy_method", dummy_method, METH_VARARGS, "Returns a dummy array"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef dummyext_module = {
    PyModuleDef_HEAD_INIT,
    "dummyext",
    NULL,
    -1,
    dummyext_methods
};

PyMODINIT_FUNC PyInit_dummyext(void)
{
    PyObject* module = PyModule_Create(&dummyext_module);
    import_array();
    return module;
}

Note that __init__.py files exist in both the rootpack and test directories. There is a trivial test file test/dummy.py that contains the from rootpack import dummyext line triggering the error.

I've tried using ext_package and changing the ext module name to just dummyext, but that results in the same error. If I remove the ext module from the nested hierarchy, it can be imported.

What is your directory structure? If I have a root directory that contains: i) your `setup.py`, ii) a directory called `rootpack` (containing an empty `__init__.py`), iii) a directory called `ext` containing `dummy.c`, then running `python setup.py install` installs the package correctly, and doing `from rootpack import dummyext` in a Python terminal (outside that directory) works for me. — Matt Pitkin, Dec 19 '22 at 16:30
The structure is as you've guessed. I can do the same thing, assuming I change my working directory to some other place (I guess it's getting confused by the local directory). But why doesn't it work in self-contained environments like tox or cibuildwheel? The goal here is to have a binary distribution, not just a local install. — Vagrant Nexus, Dec 19 '22 at 17:26
Do you have a `MANIFEST.in` file that includes `include ext/dummy.c`? — Matt Pitkin, Dec 19 '22 at 21:27
You might also want to try having `pytest-runner` as in this question https://stackoverflow.com/questions/49033350/how-to-use-cython-with-pytest — Matt Pitkin, Dec 19 '22 at 21:27
You can just have it contain the line `include ext/dummy.c`, so that it knows to include that file in the distributed package. — Matt Pitkin, Dec 19 '22 at 21:28
Unfortunately, neither of those things rectified the issue. From my reading, it looks like `MANIFEST.in` should automatically pick up the source file, since it's listed as part of an extension. — Vagrant Nexus, Dec 19 '22 at 21:57

Nested extension module cannot be imported

0 Answers0