With PyO3 how can I import a Python module once and then use it in multiple Rust threads?

Question

Using PyO3, from a long-running Rust application I want to be able to call some functions many times. Before that I need to import a few modules. How can I do the imports just once and then re-use the PyModules from other threads that call the functions? Here is a simplified example of what I'm trying to do:

let x: PyResult<&PyModule> = Python::with_gil(|py| {
    let pm = py.import("time")?;  // Only do this once
    Ok(pm)
});
let my_module = x.unwrap();
let mut handles = vec![];
for i in 0..2 {
    let t = thread::spawn(|| {
        Python::with_gil(|py| {
            // Use my_module here
        });
    });
    handles.push(t);
}
for h in handles {
    h.join();
}

The above code will not compile due to lifetime errors. How should I change it, so that I can use that shared my_module instance across my app, in different threads?

Caesar · Answer 1 · 2023-04-13T07:21:22.947

You need three things for this to work

The return value of import is an &'py PyModule. The 'py means you cant take that out of the with_gil scope. (The for<'py> here enforces that. Roughly, it says that 'py might be　anything, for example as short as the call to with_gil.) You must convert it into a Py<PyModule>, which you're free to do with what you want.
If you're planning on passing a reference to a thread (that is, a &Py<PyModule>), rust must know that the thread can't outlive the reference. Crucially, it must know that from the type/lifetime system, which isn't affected by h.join(). The solution is scoped threads. (You could also solve this by cloneing the Py in the loop before you spawn the thread.)
Finally, you can remove that pesky Py wrapper with .as_ref(py) as long as you have another GIL.

use pyo3::prelude::*;

fn main() {
    let x: PyResult<Py<PyModule>> = Python::with_gil(|py| {
        let pm = py.import("time")?; // Only do this once
        Ok(pm.into())
    });
    let my_module = x.unwrap();
    std::thread::scope(|s| {
        for _ in 0..2 {
            s.spawn(|| {
                Python::with_gil(|py| {
                    my_module.as_ref(py);
                });
            });
        }
    });
}

I do hope you're doing something other than acquiring the GIL in your threads, otherwise spawning threads would be rather pointless.

You can `clone()` the `Py` and avoid scoped threads, if needed. — Chayim Friedman, Apr 13 '23 at 06:55
Ah, indeed. Thought [cloning](https://docs.rs/pyo3/latest/pyo3/prelude/struct.Py.html#method.clone_ref) required the GIL. — Caesar, Apr 13 '23 at 07:17

With PyO3 how can I import a Python module once and then use it in multiple Rust threads?

1 Answers1