Rust Polars .par_iter() for ChunkedArray?

Question

In Rust, using Polars, I am writing a custom function to be used within apply/map. This works well:

fn capita(x: Series) -> Result<Series> {
    let y = x
            .utf8()
            .unwrap()
            .par_iter() //ParallelIterator

However, if my Series was of type f64, then this doesn't work:

fn other_fn(x: Series) -> Result<Series> {
    let y = x
            .f64()
            .unwrap()
            .par_iter() // <--- no method par_iter found for reference ChunkedArray<Float64Type>

Is there a possible workaround? Looking at utf8.rs I need something like that but for Float64Chunked type.

Many thanks

EDIT I think this workaround has worked, although turned out to be slower, AND giving an unexpected result. using par_bridge:

pub fn pa_fa(s: &mut [Series])->Result<Series>{
    let u = s[2].f64()?;
    let n = s[1].f64()?;
    let n_iter = n.into_iter();

    let c: Vec<f64> = n_iter.zip(u).par_bridge().map(
    // ignore this line   let c: Vec<f64> = n_iter.zip(u).map( 
        |(n,u)|{
            n.unwrap().powf(1.777)*u.unwrap().sqrt()
        }
    ).collect();
    Ok(Series::new("Ant", c))

`par_iter` is not yet supported for all `ChunkedArray`s. If you want parallelization you can to slice the arrays in the number of threads and process those in parallel. This gives you far larger units of work per threads. `polars_core::split_ca` might give some inspiration. Its private so don't depend on it. — ritchie46, May 08 '22 at 09:03

Claudio Fsr · Accepted Answer · 2023-03-13T18:29:59.210

The alternative I found was to collect chunked_array into a vector:

use polars::prelude::*;
use rayon::prelude::*;

fn round_series_float64(series: Series, decimals: u32) -> Series {

    let chunked_array: &ChunkedArray<Float64Type> = series.f64().unwrap();

    let vec_option_f64: Vec<Option<f64>> = chunked_array.into_iter().collect();

    let series: Series = vec_option_f64
        .into_par_iter() // rayon: parallel iterator
        //.into_iter()
        .map(|option_f64: Option<f64>|
            option_f64.map(|float64: f64| round_f64(float64, decimals))
        )
        .collect::<Float64Chunked>()
        .into_series();

    series
}

Such that:

fn round_f64(x: f64, decimals: u32) -> f64 {
    let y = 10i32.pow(decimals) as f64;
    (x * y).round() / y
}

Rust Polars .par_iter() for ChunkedArray?

1 Answers1