Python C-ext namespace mixed with regular python submodules?

Question

This question has some resemblance with:

Only with a slight twist. Here I'm not trying to mix two C-exts, but one C-ext and a regular python submodule instead.

Is there a way for a C-extension to share the module namespace between the symbols "module.so" and those present in a submodule?

My module structure looks like this:

facs/
    facs/
      __init__.py
      setup.py
      facs.so
      [*.c files]
      utils/
        __init__.py
        galaxy.py

If I remove "utils" from the hierarchy, I can import facs and see the facs.so methods:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove']

But when I put the utils submodule back and try to import the different parts, one namespace seems to mask the other (utils masks the symbols exported by facs.so):

>>> import facs
>>> dir(facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
>>> import facs.utils
>>> facs.utils.galaxy.rsync_genomes("phix")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'galaxy'
>>> from facs.utils import galaxy
>>> galaxy.rsync_genomes("phix")
'Hello world'

As you see, after dir(facs), build, query and remove are gone and galaxy does not get imported properly unless I do a from facs.utils import galaxy instead of re-using the initial import statement and simply accessing via facs.utils.galaxy.rsync_genomes() directly.

To sum up, my intended use of this module would be to do:

>>> import facs
>>> dir(facs)
['__doc__', '__file__', '__name__', '__package__', 'build', 'query', 'remove'
, 'utils'] <--- (Directly accessible from "facs")
>>> facs.utils.galaxy.rsync_genomes("phix")
'Hello world'

The (currently under development) code is in:

https://github.com/brainstorm/facs/tree/develop

In case someone wants to try it out by themselves. I'm using virtualenvs and my $PYTHONPATH seems correct:

/home/roman/.venvburrito/lib/python:
/home/roman/.virtualenvs/py27/lib/python2.7/site-packages

And the installation seems to be successful too:

cd ~/.virtualenvs/py27/lib/python2.7/site-packages/facs-2.0dev-py2.7.egg/
(py27)$ ls
EGG-INFO  facs.py  facs.pyc  facs.so  utils/

It seems that no __init__.py file is actually copied to the top level directory but touching it there or not does not affect the importing behavior described above.

Any ideas? Thanks in advance!

Did you try to rename `facs.so` to something else? Try to use a different name for the package and the `.so` — Bakuriu, Mar 01 '13 at 16:25
The `.so` file is generated when the python module is compiled/installed, I don't generate it myself... sorry if that wasn't clear :-S — brainstorm, Mar 01 '13 at 17:31

score 0 · Accepted Answer · edited Mar 04 '13 at 09:11

0

Be carefull you are importing the so file first, and not the facs package. Are you running your python console over facs package source directory ?

Try to go outside the package directory and everything will make sense.

Also, next import doesn't say that all modules behind your facs directory will be imported:

>> import facs
>> dir (facs)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']

You have to set your imports in each __init__.py file, for example if you want to export some simbols from your .so file when you import facs, you should have the following contents in your __init__.py file:

>> from _facs import function_name, function_name .....

And then, when you import facs, it will import those functions for you.

Take the same pattern for your sub-package utils.

I would also recommend to rename your facs.so file to _facs.so file to avoid name collisions between package name and module name.

edited Mar 04 '13 at 09:11

brainstorm

720
7
24

answered Mar 01 '13 at 17:32

pfreixes

439
2
6

Hello pfreixes! It's nice to see you over here! I actually put in place your suggestions, but that doesn't seem to fix my problem :_S: https://github.com/brainstorm/facs/commit/ffa1df9be3d8167c5d5ae4146bf4f8ae4273d8fe – brainstorm Mar 01 '13 at 21:48
After the commit above, I get "Import error: no module named facs", so I cannot load any of the symbols :-/ Again, thanks a lot for having a look at that! :D – brainstorm Mar 01 '13 at 21:52
Sorry but facs/__init__.py should be """from facs._facs import ...""" and it should be work ... – pfreixes Mar 04 '13 at 08:53
File "/home/roman/dev/facs/facs/__init__.py", line 1, in `from facs._facs import build, query, remove` ImportError: dynamic module does not define init function (init_facs) – brainstorm Mar 04 '13 at 09:07
I've observed that the facs/__init__.py file I just edited, does not get copied to the .virtualenv dir: $ ls ~/.virtualenvs/py27/lib/python2.7/site-packages/facs-2.0dev-py2.7-linux-x86_64.egg/ EGG-INFO _facs.py _facs.pyc _facs.so utils .... but the result is the same, the module does not get imported at all (ImportError: No module named facs). – brainstorm Mar 04 '13 at 09:23
For the record, I'm trying the `import facs` from outside the repository itself (in $HOME to be more specific). – brainstorm Mar 04 '13 at 09:29
Getting closer: `SystemError: dynamic module not initialized properly`there seems to be a problem with the `initfacs` function... – brainstorm Mar 04 '13 at 14:43
Solved!: https://github.com/brainstorm/facs/commit/44b72696a240bdc2a80cdfb2ad467fd18df6ce99 Thanks pfreixes! – brainstorm Mar 04 '13 at 15:18
When I use this exact same strategy, I get "No module named _facs". Argh, so frustrating. – Translunar Jan 13 '16 at 17:33

Python C-ext namespace mixed with regular python submodules?

1 Answers1