I have a Rust library using PyO3, and one of my functions looks like this:
#[pyfunction]
pub fn my_func<'a>(py: Python<'a>) -> PyResult<&'a PyBytes> {
let my_data: Vec<u8> = do_stuff(); // very large data here, >1GB
let result = PyBytes::new(py, &my_data[..]);
println!("Returning {} bytes to Python", my_data.len());
Ok(result)
}
From Python, I can successfully get my data:
import MyLibrary
result : bytes = MyLibrary.my_func()
I've found that the first time I call this, it works fine. The second time I call it, my library prints out the message then fails, giving me something like memory allocation of 1073741824 bytes failed -- meaning the failure is occurring in the passing of memory between them.
If I run two of these calls sequentially, I see that my RAM utilization is about 11 GiB after the first call; it spikes to >16 GiB before failing on the second call. Right now, I'm doing some testing in which I don't actually use the return data and simply overwrite it; basically:
for i in range(10):
result = MyLibrary.my_func()
print(len(result))
It seems that my data are allocated in Rust, returned to Python, then not freed. Is there something I can do (either in Rust or Python) to make sure that the memory gets freed?
