Decoupling task submission and result processing while using ExecutorService/ThreadPool

Question

I have use-case where in I should setup a scheduled job which triggers at a certain time interval. The job is to fetch workItems ( if any present) based on a certain criteria and process them. In processing these items I am trying to use and Executor with a fixedThreadPool and process them in parallel .

I would want the main thread which is a scheduled job which fetches the work items to just submit the tasks to the executor and move on to its next iteration. But my tasks also returns the result of the processing so I need to read the results of each task and handle them accordingly. Since I have to read the results I am not able end the main threads processing at submitting the tasks to the executor. The main thread is waiting (technically) for the tasks to complete and process the results and then end its execution

My Polling class - Poller.java - It keeps polling at a certain frequency

 public void startPollingProcess() {

        try {
            List<RenderingsLogEntity> extractionLogDelta = fetchWorkItems();

            if (extractionLogDelta != null && !extractionLogDelta.isEmpty()) {
                processor.processWorkItems(extractionLogDelta);
            } 

        } 
    }

My Processor class - Processor.java

// initializing the executor at some point in the class. And the thread pool will be using a worker queue with bounded capacity. If the queue is full for any reason and cant take any more tasks it will throw a RejectedExecutionException as you can see

ExecutorService executorService = new ThreadPoolExecutor(5, 5,
                0L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<>(3000))


 public void processWorkItems(List<RenderingsLogEntity> workItems) {


        List<Future<ProcessingResult>> processingResultsList = new ArrayList<>();

        try {

            // each task is submitted to the thread pool of bounded queue size. I would want the main thread to just submit these tasks. If any exception handle that. 
            for (WorkItem item : workItems) {
                processingResultsList.add(executorService.submit(() -> furtherProcessing(item)));
            }

        } catch (RejectedExecutionException e) {

           // do something in case of exception
            }
        }

     // Passing the handle on Future Results to a different method
        if (!deltaProcessingResultList.isEmpty())
            publishResults(deltaProcessingResultList);
    }


    private void publishResults(List<Future<ProcessingResult>> processingResultsList) {

        try {
            // Reading the results of each task and handling it. This is still being done by the main thread and result.get() is a blocking call.  My  polling execution  is not complete until I handle these results. But I want that to be done by someone else and I want my polling to continue 
            for (Future<ProcessingResult> result : deltaProcessingResultList) {
                ProcessingResult resultStatus = result.get();

            // handle the result accordingly
            }

        } catch (InterruptedException | ExecutionException e) {
            logger.error(e.getMessage(), e);
        }
    }

score 0 · Answer 1 · answered Aug 07 '20 at 10:21

Sounds like you want to processWorkItem and publishResult in your worker thread;

public void processWorkItems(List<RenderingsLogEntity> workItems) {
    try {
      for (WorkItem item : workItems) {
        executorService.submit(() -> {
          ProcessingResult result = furtherProcessing(item);
          publishResult(result);
        });
      }
    } catch (RejectedExecutionException e) {
       // handle exception
    }
}

Decoupling task submission and result processing while using ExecutorService/ThreadPool

1 Answers1