Executing external commands in parallel and capturing the output in an array in Rust

Question

I have the following while loop that runs generate_user_key for each of the file in the file_array, and outputs the result. I would like to parallelize this such that an array of the generated keys is returned, and the process is executed in parallel instead of sequential to make it faster.

use std::process::Command;

//file_array definition here

let mut i = 0;
while (i<100) {
  let generated_key = Command::new("generate_user_key")
                       .arg(file_array[i])
                       .output()
                       .expect("generate_user_key command failed to start");
  println!("stdout: {}", String::from_utf8_lossy(&generated_key.stdout));
  i=i+1;
}

What is the best way to implement this in rust?

what is `file_array` type in the question? – Chandan Apr 19 '22 at 10:13 — Chandan, Apr 19 '22 at 10:13

Chandan · Accepted Answer · 2022-04-19T04:39:58.607

If you want to loop over the array items using rayon then you can simply create into_par_iter and work on array items

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = arr.into_par_iter().flat_map(|value| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", value)])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", value, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

You can also use range to loop over and use the counter as array index

use std::process::Command;
use rayon::iter::{ParallelIterator, IntoParallelIterator};

fn main() {
    let arr = [1, 2, 3, 4, 5];
    let result: Vec<_> = (0..arr.len()).into_par_iter().flat_map(|idx| {
        let output = Command::new("sh")
                .args(["-c", &format!("echo {}", arr[idx])])
                .output()
                .expect("failed to execute process");
        println!("Index: {}, Output: {:?}", idx, output.stdout);
        output.stdout
    });

    println!("{:?}", result);
}

Example using `thread`

use std::thread;
use std::time::Duration;

fn main() {
    let mut threads = vec![];
    for idx in 0..arr.len() {
        threads.push(thread::spawn(move || -> Vec<_> {
            let output = Command::new("sh")
                    .args(["-c", &format!("echo -n {}", idx)])
                    .output()
                    .expect("failed to execute process");
            println!("Index: {}, Output: {:?}", idx, output.stdout);
            thread::sleep(Duration::from_millis(1));
            output.stdout
        }));
    }

    let result = threads.into_iter().flat_map(|c| c.join().unwrap()).collect::<Vec<_>>();

    println!("{:?}", result);
}

Thanks Chandan. In the case of into_par_iter, if I need to add the output to an array instead of printing it, how would I do that? Thanks — Saqib Ali, Apr 18 '22 at 21:13
@SaqibAli you can use [`map`](https://docs.rs/rayon/1.5.1/rayon/iter/trait.ParallelIterator.html#method.map) or [`flat_map`](https://docs.rs/rayon/1.5.1/rayon/iter/trait.ParallelIterator.html#method.flat_map) to collect the result as `vec` or `array` — Chandan, Apr 19 '22 at 03:51
When I run the first one, I get "the trait `ParallelIterator` is not implemented for `&str`" — Saqib Ali, Apr 19 '22 at 04:50
@SaqibAli please can you share how you are trying the first example in your question — Chandan, Apr 19 '22 at 05:02
@SaqibAli it seems you are trying to use the example in data of [`&str`](https://doc.rust-lang.org/std/primitive.str.html) type which does not implement [`iterator`](https://doc.rust-lang.org/std/iter/trait.Iterator.html#) trait, for string you can check [here](https://docs.rs/rayon/1.5.1/rayon/str/trait.ParallelString.html). — Chandan, Apr 19 '22 at 05:14

Jmb · Answer 2 · 2022-04-19T14:16:23.723

This should be easy to do with rayon. E.g. something like this (untested since I don't have your generate_user_key):

use rayon::prelude::*;
let keys = (0..100).into_par_iter().map (|_| {
        Command::new("generate_user_key")
            .arg(file_array[i])
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();

or better:

use rayon::prelude::*;
let keys = file_array.par_iter().map (|f| {
        Command::new("generate_user_key")
            .arg(f)
            .output()
            .expect("generate_user_key command failed to start")
            .stdout
    })
    .collect::<Vec<_>>();

Locke · Answer 3 · 2022-04-18T11:52:18.817

-1

When all else fails, throw threads at the problem. It almost certainly isn't the correct approach, but it works.

let mut join_handles = Vec::new();

for _ in 0..100 {
    join_handles.push(thread::spawn(|| {
        let generated_key = Command::new("generate_user_key")
                              .arg(file_array[i])
                              .output()
                              .expect("generate_user_key command failed to start");

        String::from_utf8_lossy(&generated_key.stdout)
    }));
}

let outputs = join_handles.into_iter().map(Result::unwrap).collect::<Vec<_>>();

Edit: The correct solution is probably using Command::spawn to start the processes without blocking. The OS can then handle running them in parallel and you can then collect the outputs.

edited Apr 18 '22 at 11:52

answered Apr 18 '22 at 11:47

Locke

7,626
2
21
41

Maybe you should add code on how to use `Command::spawn` instead of how to start threads, since you say the former is the correct solution, which the latter is the plan B. – jthulhu Apr 18 '22 at 12:01
Where is the i for file_array[i] getting set? – Saqib Ali Apr 18 '22 at 12:58

Executing external commands in parallel and capturing the output in an array in Rust

3 Answers3

Example using thread

Example using `thread`