What are the differences between these 4 methods of returning `bytes` from Rust to Python using `pyo3`?

Question

I'm new to pyo3 and a beginner in Rust. What I'm trying to do is to return from Rust slice to Python bytes. I read both about the type conversion and memory management in the pyo3 documentation, but I still feel a bit lost.

This is what I have right now:

#[pyclass]
pub struct PyRust {}

#[pymethods]
impl PyRust {
    fn test_bytes_1(&self, py: Python) -> Py<PyAny> {
        let v = vec![1, 2, 3];
        let res: Py<PyAny> = v.as_slice().into_py(py);
        res
    }

    fn test_bytes_2(&self) -> Py<PyAny> {
        let v = vec![1, 2, 3];
        let res: Py<PyAny> = Python::with_gil(|py| v.as_slice().into_py(py));
        res
    }

    fn test_bytes_3(&self, py: Python) -> Py<PyBytes> {
        let v = vec![1, 2, 3];
        let res: Py<PyBytes> = PyBytes::new(py, v.as_slice()).into();
        res
    }

    fn test_bytes_4(&self) -> Py<PyBytes> {
        let v = vec![1, 2, 3];
        let res: Py<PyBytes> = Python::with_gil(|py| PyBytes::new(py, v.as_slice()).into());
        res
    }
}

I see that all of them when called from Python return what I expect:

In [3]: x.test_bytes_1()
Out[3]: b'\x01\x02\x03'

In [4]: x.test_bytes_2()
Out[4]: b'\x01\x02\x03'

In [5]: x.test_bytes_3()
Out[5]: b'\x01\x02\x03'

In [6]: x.test_bytes_4()
Out[6]: b'\x01\x02\x03'

Can you explain to me what's the difference? What is the best way? Do they have different costs?

If I use %timeit I have these:


In [4]: %timeit x.test_bytes_1()
2.05 µs ± 18.9 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [5]: %timeit x.test_bytes_2()
1.77 µs ± 7.85 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [6]: %timeit x.test_bytes_3()
1.97 µs ± 14.2 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)

In [7]: %timeit x.test_bytes_4()
1.78 µs ± 14 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

So it seems like using Python::with_gil is faster, but why?

Also, how Py<PyAny> is converted back to bytes in the same way as Py<Bytes>? Does PyAny has any "remembrance" that it has been created from a PyBytes? Or something different happens there?

score 0 · Answer 1 · answered Aug 19 '22 at 12:09

Can you explain to me what's the difference?

Looking at the code, there doesn't really seem to be a difference, they essentially do the same thing through slightly different path and will largely delegate to one another e.g.

v.as_slice().into_py(py)

should call this:

impl<'a> IntoPy<PyObject> for &'a [u8] {
    fn into_py(self, py: Python<'_>) -> PyObject {
        PyBytes::new(py, self).to_object(py)
    }
}

So it seems like using Python::with_gil is faster, but why?

Could be because the optimiser gets better insight into its operation, and so it's able to generate more efficient code: for functions 1 and 3, the gil has to be managed / acquired by pyo3's generic glue code, so the compiler has no insight into it (it could also be a slower codepath overall?)

What are the differences between these 4 methods of returning `bytes` from Rust to Python using `pyo3`?

1 Answers1