4

Trying to build modules hierarchy with pyo3 and this code

pub mod types;
pub mod sources;

use pyo3::prelude::*;
use pyo3::wrap_pymodule;

use sources::file::{find_days, read_many, read_one};


#[pymodule]
fn file(_py: Python, m: &PyModule) -> PyResult<()> {

    #[pyfn(m, "find_days")]
    fn find_days_py(_py: Python, dir: String) -> PyResult<Vec<String>> {
        let out = find_days(&dir)?;
        Ok(out.iter().map(|x| String::from(x.to_str().unwrap())).collect())
    }

    Ok(())
}

#[pymodule]
fn sources(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pymodule!(file))?;
    Ok(())
}

#[pymodule]
fn cstuff(_py: Python, m: &PyModule) -> PyResult<()> {
    m.add_wrapped(wrap_pymodule!(sources))?;
    // m.add("__path__", vec![""])?;
    Ok(())
}

Code builds and works fine, except that when I try to import it, I'm getting this error

In [1]: import cstuff.sources.file                                                                      
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-97430c5317d9> in <module>
----> 1 import cstuff.sources.file

ModuleNotFoundError: No module named 'cstuff.sources'; 'cstuff' is not a package

From what I understand from python documentation, module is a package if it have __path__ attribute.

I can add that with m.add("__path__", vec![""])?;, after this import works as expected but only when python is launched from the dir with .so file, which is expected.

The problem is that I don't know __path__ in advance. How to fix that, is there any way to force python to set __path__?

cvb
  • 337
  • 2
  • 8
  • In the end, you are not creating a package, but rather a module that holds a reference to another module as a member variable. If you look at the exported symbols of your `.so` file using `nm foo.so | grep ' T '`, you'll see that there is one `PyInit_` function for each module, so you could technically symlink the library as each of those and import them from the top level right away. In my experience, extension modules are usually not exposed directly to the Python end user but rather placed inside a Python package, that constitutes the public interface. – FallenWarrior Jan 09 '20 at 20:10
  • Well, that is true, symlinking worked, that was surprising. But what I'm trying to achieve is to mimic existing pure-python `sources.*` modules hierarchy in rust in the least painful way. – cvb Jan 09 '20 at 21:35
  • The problem is that the module names aren't qualified. So even if you have a wrapper package (e.g. `test`) + `__init__.py` around your extension module, trying to import a module by its "full name" (like `import test.foo`) will still fail because the module name is just `foo`, and not `test.foo`. – FallenWarrior Jan 09 '20 at 22:14
  • i don't get that. I just made dirs `cstuff/sources` each with `__init__.py` and put my `file.cpython-37m-x86_64-linux-gnu.so` and now `import cstuff.sources.file` works as expected, and I thought that this is the way this extensions expected to be used. But what if I want to have `cstuff.sources.file` and `cstuff.sources.zmq` for example? Rust build only one big shared library, I can do symlinking or just have two copies of this shared library with appropriate names, but that looks ugly. – cvb Jan 10 '20 at 07:55
  • You could export a single module from your Rust library and then use thin Python wrappers to build a package hierarchy out of that, i.e. have each Python module import the names it needs from the extension module using `from` imports, which automatically re-exports them under the module the import is in. If you want to re-export nested properties, you can import them and assign them to a module-level variable so that they are exported under the name of that variable. – FallenWarrior Jan 11 '20 at 00:09
  • I thought about that, but this require you to export function in rust, and then reexport it in python, and I'm too lazy for that. I'll probably stick with having multiple extensions, like this guys did https://github.com/ManifoldFR/point-process-rust/blob/master/pylib/setup.py#L41. Thanks anyway. – cvb Jan 14 '20 at 18:29

0 Answers0