Compiling pyx files with dependencies in different packages

Question

I am having problems compiling cdef-ed types in different packages and I couldn't find an explanation in cython docs.

I have this setup.py in the root of my python src tree:

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

setup(
   cmdclass = {'build_ext': build_ext},
   ext_modules = [ 
      Extension("flink.pytk.defs.FragIdx", 
         sources = ["flink/pytk/defs/FragIdx.pyx"]),
      Extension("flink.pytk.fragments.STK_idx", 
         sources = ["flink/pytk/fragments/STK_idx.pyx"])
      ]   
)

FragIdx is a cdef-ed type, defined in flink/pytk/defs/FragIdx.pyx:

cdef class FragIdx:
   cdef public FragIdx parent
   cdef public FragIdx root
   cdef public tuple label
   ...

And STK_idx is an extension of FragIdx, defined in flink/pytk/fragments/STK_idx.pyx:

from flink.pytk.defs.FragIdx import FragIdx
cdef class STK_idx(FragIdx):
   ...

When I try to compile using the setup.py listed at the beginning of the post, FragIdx is compiled all right, but when it comes to STK_idx I get the following error message:

flink/pytk/fragments/STK_idx.pyx:5:5: 'FragIdx' is not a type name

Please note that the root directory of my source tree is listed in $PYTHONPATH.

I would really appreciate if anyone could shed any light on this, thanks a lot!

Daniele

score 12 · Accepted Answer · edited Mar 25 '11 at 12:52

Oh, well, for those having a similar problem, it looks like maybe I found the answer.

I was expecting python to automatically scan the symbols compiled into the shared library FragIdx.so, instead it looks like this information must be provided explicitly as a .pxd file (which becomes a C header file after Cython is run).

There are basically two steps involved in the process:

Creation of a definition (.pxd) file for the superclass;
Importing of the the superclass definition via cimport (as opposed to import) in the subclass module.

So, to make it more general.

Suppose that you have defined your cdef-ed type A in module pkg1.mod1. Then you cdef a type B in pkg2.mod2 that subclasses A.

Your directory structure would look something like this:

pkg1/
  mod1.pyx
  mod1.pxd
pkg2/
  mod2.pyx
  mod2.pxd

In pkg1/mod1.pxd you would have, say:

cdef class A:
  cdef int a
  cdef int b

And in pkg1/mod1.pyx you would provide the methods of your class. In pkg2/mod2.pxd, you would have:

from pkg1.mod1 cimport A  #note "cimport"!!
cdef class B(A):
  cdef ... # your attributes here

And again, in pkg2/mod2.pyx you would have to cimport the A symbol again:

from pkg1.mod1 cimport A #note "cimport"!!
cdef class B(A):
  ... # your methods here

Interestingly enough, if you just want to use A in your python code, as opposed to using it to define a subtype, the definitions file mod1.pxd is not needed. This is related to the fact that when creating an extension type you need the definitions to be available for the C compiler, whereas you don't have this problem when running python code, but since it is not very intuitive maybe it's important to point it out.

This information is actually available in the Cython docs, though maybe it could be a little bit more explicit.

Hope this information can save some to someone.

Compiling pyx files with dependencies in different packages

1 Answers1