0

I am working implementing parallel batch processing in Java. My idea is let's say there is array of 12 elements, split the array in to 3 arrays of size 4 and get the sum of all elements of each array and print the result. I tried implementing solution using ExecutorCompletionService. Below is code I have written.

Issue here is I am still looping for partitions and getting the sum.

Is there is any way to parallelize the process of getting sum for all partitions(3 arrays)?

My intention is to get sum of each partition (array) parallely instead of looping over all partitions one by one.

My goal is to split the array in to multiple partitions. Process all partions parallely.

Please help.

import java.util.*;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutorCompletionService;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;

import com.google.common.collect.Lists;

public class TestMultiThread {
    public static void main(String[] args) throws InterruptedException {
        System.out.println("Hello test multi threading");
        List<Integer> values = new ArrayList<>(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12));
        List<List<Integer>> partition = Lists.partition(values, 4);

        final ExecutorService pool = Executors.newFixedThreadPool(10);
        final ExecutorCompletionService<Integer> completionService = new ExecutorCompletionService<>(pool);
        List<Future<Integer>> futures = new ArrayList<>();
        for (List<Integer> value:partition) {
            futures.add(completionService.submit(getArraySize(value)));
        }

        for(Future<Integer> future: futures) {
            future = completionService.take();
            try {
                Integer integer = future.get();
                System.out.println(integer);
            } catch (Exception e) {
                System.out.println("Failed");
            }
        }

        futures.forEach(future -> future.cancel(true));
    }


    private static Callable<Integer> getArraySize(List<Integer> value) {
        Callable<Integer> callable = () -> {
            int sum = 0;
            for (int i : value) sum += i;
            return sum;
        };
        return callable;
    }
}
sudhir
  • 219
  • 5
  • 17
  • I don't see any problem in the implementation. If you are using `ExecutorCompletionService`, you will have to `take()` the results one-by-one. That is its design. Also, submssion of tasks to it also has to be happen in loop. Only the processing of each submitted will happen in parallel. I think the code is quite OK. – Sree Kumar Oct 14 '21 at 06:53
  • Makes sense. Is it possible to make parallel calls, one for each partition using any other Java library without using ExecutorCompletionService. – sudhir Oct 14 '21 at 06:59
  • My goal is aplit the array in to multiple partitions. Process all partions parallely. – sudhir Oct 14 '21 at 07:04
  • There are 4 actions: (a) Splitting, (b) Submission of tasks, (c) Processing of tasks and (d) Reading results. (a) and (b) must happen in sequence since the array that has the initial set of data has to be partitioned in only one thread, to avoid erroneous partitioning. Then, that (c) happens concurrently is the lookout of `ExecutorService`. (d) is where you have choice (if you have hundreds of parallel tasks, then you may think of `take()` in multiple threads). As an aside, submission of tasks is not so costly that we attempt to parallelize it, I think. – Sree Kumar Oct 14 '21 at 07:10
  • https://stackoverflow.com/questions/18162863/how-to-run-different-methods-parallely This solution might work for me. Will create 4 callable objects 1 for each partition and use executor service to call the task list and aggregate the final result. Do you think this is better solution fo ruse case? – sudhir Oct 14 '21 at 07:30

0 Answers0