0

I want to create the following module and object structure:

main_module
|
`--sub_module
   |
   +--ObjectOne
   |
   `--sub_sub_module
      |
      `--ObjectTwo

When I only had main_module.sub_module everything worked as it should, as I created an empty __init__.py inside the main_module folder (since main_module does not have any objects at the moment), and I placed sub_module.so next to it.

However, when I tried to create two separate c modules to make main_module.sub_module.sub_sub_module work (that is sub_module.so and sub_sub_module.so) and then I added two __init__.pys which both of them imported their extension modules relative to them, then I got into ImportErrors (as wrappers around the undefined symbol error "raised" from the shared libraries), because sub_sub_module needs a few of the C-level definitions from sub_module. Which makes me think, it would be way more easier, to create a single shared library which somehow creates "virtual" modules inside it, instead of the unnecessary library linking..

So my question is: Is it possible? If so, how? Or, is there a better way to achieve what I'm looking for?

Peter Varo
  • 11,726
  • 7
  • 55
  • 77
  • Could you elaborate a little on how you implemented all this; or better post some code? – tynn Oct 10 '15 at 19:50
  • @tynn it is quite large to post the whole thing here, but here is the repo: https://github.com/kitchenbudapest/hackathon/tree/master/fw/wrappers/python -- if you go a few folders up, in `kb` there is the C library I'm wrapping, and the link I provided is the python wrapper part only – Peter Varo Oct 14 '15 at 11:09
  • ***um.. why the -1?*** – Peter Varo Oct 16 '15 at 08:39

1 Answers1

2

I think I got your problem now. For a long and detailed answer you should read Providing a C API for an Extension Module.

Portability therefore requires not to make any assumptions about symbol visibility. This means that all symbols in extension modules should be declared static, except for the module’s initialization function, in order to avoid name clashes with other extension modules [...]. And it means that symbols that should be accessible from other extension modules must be exported in a different way.

To achieve this you should use Capsules. For these every function you want to use in another module should be stored inside a void* array. Then you create a capsule with PyCapsule_New(void*, const char*, PyCapsule_Destructor) using the array as first argument. This object must be added to your sub_module. Then you can call PyCapsule_Import(const char*, int) in your sub_sub_module to import the array and access the functions you need.

tynn
  • 38,113
  • 8
  • 108
  • 143
  • I'm already familiar with `Capsule`s -- as You can see, I use them for the `enum` values (PIN indices). However I've never thought of using them like you have suggested.. My other solution -- which I think can be better than the one you have provided -- is to create 3 shared libraries: 1) for common-data, 2) for sub_module, 3) for sub_sub_module => in this case there is no "Python overhead" where it is not necessary. IMHO `Capsule`s are there to make opaque C-level data passing *by the user* -- but here, the data is not necessarily used by the user directly. (+1 for the effort though!) – Peter Varo Oct 16 '15 at 08:23
  • 1
    @PeterVaro it's from the official documentation. I was just paraphrasing. In my understanding `Capsule`s should always be used instead of exporting symbols and python overhead is quite insignificant since you're only importing it once. But if it's not python related data and functions I might also link to a "pure" C library instead. – tynn Oct 16 '15 at 09:18