3

I am trying to do something where I have a little logic in groupby. Consider this working code for example:

use itertools::Itertools;
let counts = vec![Some(1), Some(1), None, None, Some(1), Some(1), Some(3), Some(3)];
let vals = vec![1, 3, 2, 2, 1, 0, 1, 2];
let mut hm1: HashMap<u32,u32> = HashMap::new();
let groups = &(counts.into_iter().zip(vals.into_iter())).group_by(|(d1,_d2)| *d1);

groups.into_iter()
.map(|(key, group)| {
    group.into_iter()
    .map(|(count, val)| {
        hm1.entry(val).and_modify(|e| *e += 1).or_insert(1);
        }).collect::<Vec<_>>();
    }).collect::<Vec<_>>();

Here, I am essentially creating a histogram in HashMap. Please ignore the fact that in this particular case I do not actually need to groupby to get my HashMap. It's illustrative.

Now, I am trying to understand how I can parallelize this type of code with Rayon. Upon searching, I found this thread that says groupby in itertools cannot be used with Rayon. Reading the Rayon documentation, I could not find any way to achieve this. Note that I don't particularly care about using itertools, so any other solution that parallelizes a groupby in rust would work.

ste_kwr
  • 820
  • 1
  • 5
  • 21
  • 1
    Note that [`Itertools::group_by`](https://docs.rs/itertools/latest/itertools/trait.Itertools.html#method.group_by) only groups _consecutive_ elements, which is inherently non-parallelizable. You can however `collect` into a `HashMap` if you want to also group non-consecutive items. – Jmb Jul 13 '22 at 06:49
  • I'm currently sorting the elements before grouping to ensure that it works. I was trying to avoid a collect OP as theoretically it should not be needed. But maybe it is faster than sort? Can I par_iter a hashmap with rayon? – ste_kwr Jul 13 '22 at 13:04
  • 1
    You will have to benchmark in order to know which is faster, but yes you can [collect a `ParallelIterator` into a `HashMap`](https://docs.rs/rayon/latest/rayon/iter/trait.FromParallelIterator.html#impl-FromParallelIterator%3C(K%2C%20V)%3E-for-HashMap%3CK%2C%20V%2C%20S%3E). – Jmb Jul 13 '22 at 13:08

0 Answers0