Implement rayon `as_parallel_slice` using iterators

Question

I have a small problem of my own:

extern crate rayon;
use rayon::prelude::*;

#[derive(Debug)]
struct Pixel {
    r: Vec<i8>,
    g: Vec<i8>,
    b: Vec<i8>,
}

#[derive(Debug)]
struct Node{
    r: i8,
    g: i8,
    b: i8,
}

struct PixelIterator<'a> {
    pixel: &'a Pixel,
    index: usize,
}

impl<'a> IntoIterator for &'a Pixel {
    type Item = Node;
    type IntoIter = PixelIterator<'a>;

    fn into_iter(self) -> Self::IntoIter {
        println!("Into &");
        PixelIterator { pixel: self, index: 0 }
    }
}

impl<'a> Iterator for PixelIterator<'a> {
    type Item = Node;
    fn next(&mut self) -> Option<Node> {
        println!("next &");
        let result = match self.index {
            0 | 1 | 2 | 3  => Node {
                r: self.pixel.r[self.index],
                g: self.pixel.g[self.index],
                b: self.pixel.b[self.index],
            },
            _ => return None,
        };
        self.index += 1;
        Some(result)
    }
}

impl ParallelSlice<Node> for Pixel {
    fn as_parallel_slice(&self) -> &[Node] {
        // ??
    }
}

fn main() {
    let p1 = Pixel {
        r: vec![11, 21, 31, 41],
        g: vec![12, 22, 32, 42],
        b: vec![13, 23, 33, 43],
    };

    p1.par_chunks(2).enumerate().for_each(|(index, chnk)| {
        for (cindex, i) in chnk.into_iter().enumerate(){
            println!("{:?}, {:?}", (index*2)+cindex, i);   
        }
    });
}

playground

Basically I wanted to use rayon's per_chunk function and it requires me to have to implement ParallelSlice trait. I am wondering what should go into the as_parallel_slice function so that I can get output as (order doesn't matter):

0 Node { 11, 12, 13} 
1 Node { 21, 22, 23}
2 Node { 31, 32, 33}
3 Node { 41, 42, 43}

One more stupid question is as_parallel_slice bounds the Trait to return a slice, according to my understanding under this scenario I need to have the full data available beforehand? Since I am working with DNA sequences (which can be a lot of data) I guess I should fall back to using crossbeam and iterators instead of slice based parallelization through rayon, or is their a better way to do the same?

score 1 · Accepted Answer · answered Jun 29 '18 at 11:56

1

You can't create a slice of Nodes unless you have a chunk of contiguous memory containing just Nodes. But you don't have that; the data from each Node is lazily copied from bits of data stored in three separate Vecs.

The most obvious way to create a slice would be to first create all the Nodes in a Vec<Node> and then make a slice of that. However, I would suspect that this is exactly what you don't want to do.

answered Jun 29 '18 at 11:56

Peter Hall

53,120
14
139
204

Just to elaborate for Avi's benefit, changing `p1.par_chunks(2)` to `p1.into_iter().collect::>().par_chunks(2)` will give the output you want but will be a complete duplication of the data. – user25064 Jun 29 '18 at 13:36
@user25064 It might be even worse than that. You can't create a vector and immediately return a slice of it. He would have to have the vector owned by `Pixel` or something else that lives as long as the slice is needed. – Peter Hall Jun 29 '18 at 13:43
Thanks @PeterHall and user25064 for the answers, you got it right, the data is too big to duplicate and would bump up the overall resource usage. I guess I'll fall back to Crossbeam based parallelization. I thing I have to work on understand the difference between the use cases for rayon and crossbeam. Thanks again, cheers! – Avi Srivastava Jun 29 '18 at 19:55

Implement rayon `as_parallel_slice` using iterators

1 Answers1

Linked