How to nest c-api extensions in a Python module and use them in that module?

Question

I want to create a Python 3 package called "mypack" that can be installed with pip. It has some c-extensions and some python code that calls those extensions. Here is my directory structure:

setup.py
mypack/__init__.py
mypack/mypack.py
foopkg/foo.cpp

The setup.py file has the following code:

from setuptools import setup, Extension

PACKAGE_NAME = 'mypack'

module = Extension('foo',
                language = "c++",
                sources = ['foopkg/foo.cpp'])

setup(name=PACKAGE_NAME,
      version='1.0',
      packages=[PACKAGE_NAME],
      ext_package=PACKAGE_NAME,
      ext_modules=[module],
      include_package_data=True)

I adapted this code from another related question where the user wanted to import the extensions using something like mypack.xxx, as I do. In the c-api extension, I have successfully compiled it and made it work as a stand-alone extension module, but I am having trouble incorporating it into a larger package. It defines two functions make_array and print_array. For brevity I removed the function code and just included the stuff that Python needs:

...

static PyMethodDef FooMethods[] = {
    { "make_array", make_array, METH_VARARGS, "Put number in array"},
    { "print_array", print_array, METH_VARARGS, "Print number from array"},
    { NULL, NULL, 0, NULL}
};

static struct PyModuleDef foomodule = {
    PyModuleDef_HEAD_INIT,
    "foo",
    "Make/Print array",
    -1,
    FooMethods
};

PyMODINIT_FUNC PyInit_foo(void)
{
    return PyModule_Create(&foomodule);
}

I want to be able to import this extension within the package to use it (this is mypack.py):

import mypack.make_array
import mypack.print_array

def dostuff():
    array = make_array(10)
    print_array(array)

Lastly, my __init__.py file contains from .mypack import dostuff.

However, when I install with pip, and try to run a test script that imports mypack it complains about the imports in mypack.py, whether I use foo.xx or mypack.xx. I have built other packages with a nested structure that use code from other python files in the module using __init__.py and using imports. But I am a bit baffled about how to do this with c-extensions.

score 0 · Accepted Answer · edited Sep 16 '17 at 12:56

0

The fact that the module is compiled from C makes no difference to how you write the import statement. The line...

import mypack.make_array

tries to import a module named "make_array" found in a package named "mypack". That's not what you have. You have a module named "foo" in a package named "mypack", and two objects (methods) in that module named "make_array" and "print_array."

It's much simpler if you remove everything from __init__.py and make it work that way. You can add stuff to the init file later if you want to improve the syntax of your import statements. You want...

from mypack.foo import make_array, print_array

or

import mypack.foo as MYPACK

In the first case the methods are now in the global namespace; in the second case you access them as MYPACK.make_array and MYPACK.print_array.

Also I see a problem with naming your C-extension file mypack and also naming your python file mypack.py. There can be only one module with that name in a package. One solution is to prefix an underscore to the C file name. Then in mypack.py you write, for example...

from ._mypack import make_array

This puts make_array into the namespace of myarray, and client code doesn't have to know, or care, that it's written in C.

edited Sep 16 '17 at 12:56

user5915738

450
5
12

answered Sep 15 '17 at 23:29

Paul Cornelius

9,245
1
15
24

Thanks for the clarification between packages and module names! I played around with your suggestion a bit. Correct me if I am wrong, but it seems that the c-api code creates the module `foo`, which I then attribute to the package `mypack` in setup.py. So when I do `from mypack.foo import make_array, print_array` in `mypack.py` it works. However, doing `from mypack.mypack import make_array, print_array` doesn't work. Now that I understand what is going on with naming I should be able to fix this in my main code. You might want to edit your answer though so it uses `from mypack.foo`. – user5915738 Sep 16 '17 at 00:19
I will edit my answer but I don't quite understand yet. After you compile foo.cpp you will get a file named foo.pyd. Normally it would be in the same directory as foo.cpp. When you import it into a python program, its full import name depends on where the file is located within the directory structure. Do you copy or move the file to the mypack directory after compiling it? Or does your setup file do that? I'm not clear what you mean by "attribute to the package mypack." As far as I know, the only way things work is if the physical file location and the import statement agree. – Paul Cornelius Sep 16 '17 at 00:43
Okay, ya, maybe I am mistaken. I am still fairly new at making modules. Okay, that is interesting, so I checked and pip install made the `.pyc` and `__init__` files under a `__pycache__` directory in `mypack` not `foopkg`. `foopkg` is where the cpp file is. – user5915738 Sep 16 '17 at 00:53
The shared library file is also under `mypack`. – user5915738 Sep 16 '17 at 01:01
I've done lots of extension modules but never wrote a pip install setup script. So we're getting a bit outside my expertise here, unfortunately. But perhaps you can take it from here, or re-ask a more targeted question. – Paul Cornelius Sep 16 '17 at 01:11
I think pip did it automatically. I'll just have to go through their documentation to see how it compiles extensions. Thanks for the help! – user5915738 Sep 16 '17 at 01:13

How to nest c-api extensions in a Python module and use them in that module?

1 Answers1