python 3.x C extension module and submodule

Question

How do I make a C extension for python 3.x when a module has sub-modules? For example, I have a file called pet.c:

#include <Python.h>

PyObject* CatMeow(PyObject* self) {
    return PyUnicode_FromString( ">*<" );
}

static PyMethodDef CatFunctions[] = {
    {(char*) "meow", (PyCFunction) CatMeow, METH_NOARGS, NULL},
    {NULL, NULL, 0, NULL}
};

static PyModuleDef CatDef = {
    PyModuleDef_HEAD_INIT, "cat", "cat ext", -1, CatFunctions,
    NULL, NULL, NULL, NULL
};

PyMODINIT_FUNC PyInit_cat(void) {
    return PyModule_Create(&CatDef);
}

static PyModuleDef PetDef = {
    PyModuleDef_HEAD_INIT, "pet", "pet ext", -1, NULL,
    NULL, NULL, NULL, NULL
};

PyMODINIT_FUNC PyInit_pet(void) {
    PyObject* p = PyModule_Create(&PetDef);
    PyObject* c = PyInit_cat();
    Py_INCREF(c);
    PyModule_AddObject( p, "cat", c );
    return p;
}

When I build it with the following setup.py:

from distutils.core import setup, Extension

setup( 
    name='pet', 
    version='0.0', 
    ext_modules=[Extension('pet', ['pet.c'])]
)

I can see

>>> import pet
>>> pet.cat.meow()
'>*<'

or

>>> from pet import cat
>>> cat.meow()
'>*<'

which is as intended, but when I try

>>> from pet.cat import meow

I have a ModuleNotFoundError saying ... No module named 'pet.cat'; 'pet' is not a package, and if I try

>>> from pet import cat
>>> from cat import meow

I have a ModuleNotFoundError saying ... No module named 'cat'. But if I check the type of cat

>>> type(cat)
<class 'module'>

which says it is a module.

How do I make this work? Adding a module object to another module used to work well in python 2.7. Is it not supposed to work in python3 due to absolute import style? Or do I have to work with multi-phase initialisation as described in PEP 489?

`from pet.cat import meow` is importing a package, you do not have a package see https://docs.python.org/3/tutorial/modules.html#packages for it to work you would need a folder named `pet` and the module named `cat` — Xantium, Feb 09 '18 at 13:37
@Simon, yes I can see that pet is not a package. But within a C extension, what does it mean by creating a folder? The link you provided explains when I work with normal python files but not C extensions. — hyunacrest, Feb 09 '18 at 13:45
Make your dictionary structure as follows: `pet/cat.pyd` make sure to remove pet from the extension, then `from pet.cat import meow` will work — Xantium, Feb 09 '18 at 13:48
@Simon, I don't get it ... When I run python3 setup.py build-ext --inplace then all it creates is a shared object (in linux .so file). No such thing as pyd file and folders. When I install this with pip3 install ., then it copies the shared object to the site-packages with egg info. — hyunacrest, Feb 09 '18 at 13:58
@Simon, this is not a python module, it is a python C extension module. It is a way to extend python with C language ... — hyunacrest, Feb 09 '18 at 19:46
But an extension is a module so you will still need an `__init__.py` file. https://docs.python.org/2/extending/extending.html — Xantium, Feb 09 '18 at 20:15

score 2 · Accepted Answer · answered Feb 10 '18 at 21:27

Regarding the first error which complains about pet not being a package. If pet is there only to provide a parent for cat, there is an easy way to turn it into a package: remove all the pet related code from pet.c and use ext_package in setup.py

from distutils.core import setup, Extension

setup( 
    name = 'pet',
    version = '0.0',
    ext_package = 'pet',
    ext_modules = [Extension('cat', ['pet.c'])]
)

Running the above will create a directory called 'pet' and a shared library of which name starts with 'cat'. This effectively creates a namespace package -- there are two types of packages, regular and namespace, and the latter is the one without requiring __init__.py (see PEP 420 for details). From outside of pet, you can do

>>> from pet import cat
>>> from pet.cat import meow 
>>> meow()
'>*<'
>>> cat.meow()
'>*<'

The reason you can't do from cat import meow is because the fully qualified name of the module is 'pet.cat' not 'cat', which you can confirm it from cat.__name__. If you are running the interpreter inside of the directory pet, then you can do from cat import meow.

Thanks for your answer. What if the pet contains extension objects? What if there are several submodules? — hyunacrest, Feb 12 '18 at 13:17
?? You can create an extension per each submodule in which case you will end up having multiple shared libraries in the pet folder. Alternatively, you can export multiple PyMODINIT_FUNCs in one shared library and use multiple symlinks as described in PEP 489. For example, if you declared cat and dog in pet.c, you can use the same setup.py as above to create cat.xxx.so in the pet folder and then create a symlink dog.xxx.so to it. For creating a package, I am not sure if you can set __path__ without creating an actual path. — Hyungsok, Feb 13 '18 at 21:42

python 3.x C extension module and submodule

1 Answers1

Linked