Java concurrent tasks implementation

Question

My problem is this: I can have a maximum of three concurrent tasks running. These tasks can process 1 to 100 jobs simultaneously. I have many threads constantly submitting single jobs, and I want to respond to them as fast as possible. The time taken to process 100 jobs in one task is the same as it takes to process 1 job in one task. Jobs come in at random intervals. Threads that submit jobs need to block until the job is done, or a timeout is hit. Responding quickly to the threads submitting jobs is the driver here.

So my current logic is this: If there are < 3 tasks running, and a job arrives, create a new task to process just that job on it's own. If there are 3 tasks running, put the job in a queue and wait until another task finishes, then take all the jobs from the queue (limit 100) and create a task to process all of them.

I'm just not quite sure the best way to set this up in Java. I created a simple semaphore versions which works fine but does not take advantage of the ability to submit job simultaneously together. How best should I expand this to fully meet my requirements? (there is no requirement to use a semaphore, it's just what I have so far).

private static final Semaphore semaphore = new Semaphore(3);

public static Response doJob(Job job) throws Exception
{ 
    final boolean tryAcquire = this.semaphore.tryAcquire(this.maxWaitTime, TimeUnit.MILLISECONDS);

    if (tryAcquire)
    {
        try
        {
            return doJobInNewTask(job); // we'd actually like to do all the jobs which are queued up waiting for the semaphore (if there are any)
        }
        finally
        {
            this.semaphore.release()
        }       
    }
}

Answer 1

You could use an Executor service with a fixed-size thread pool:

class ExecutorExample {
    private final static ExecutorService executorService;
    private final static long maxWaitTime = 5000;

    static {
        executorService = Executors.newFixedThreadPool(3);
    }

    private static class Response {}
    private static class Job {}

    public static Response doJob(final Job job) throws Exception {
        final Future<Response> future = executorService.submit(
            new Callable<Response>() {
                @Override
                public Response call() throws Exception {
                    return doJobInNewTask(job);
                }
            }
        );
        try {
            // get() blocks until the task finishes.
            return future.get(maxWaitTime, TimeUnit.MILLISECONDS);
        }
        catch (final TimeoutException e) {
            // we timed out, so *try* to cancel the task (may be too late)
            future.cancel(/*mayInterruptIfRunning:*/false);
            throw e;
        }
    }

    private static Response doJobInNewTask(final Job job) {
        try { Thread.sleep(maxWaitTime / 2); }
        catch (final InterruptedException ignored) {}
        return new Response();
    }

    public static void main(final String[] args) {
        final List<Thread> threads = new ArrayList<>();

        for (int i = 0; i < 10; i++) {
            final Thread t = new Thread() {
                @Override
                public void run() {
                    try {
                        System.out.println(doJob(new Job()));
                    }
                    catch (final Exception e) {
                        System.out.println(e.getClass().getSimpleName());
                    }
                }
            };
            threads.add(t);
            t.start();
        }

        for (final Thread thread : threads) {
            try { thread.join(); }
            catch (final InterruptedException ignored) {}
        }

        System.out.println("Done!");
    }
}

Output:

ExecutorExample$Response@1fe4169
ExecutorExample$Response@9fdee
ExecutorExample$Response@15b123b
ExecutorExample$Response@bbfa5c
ExecutorExample$Response@10d95cd
ExecutorExample$Response@131de9b
TimeoutException
TimeoutException
TimeoutException
TimeoutException
Done!

One potential issue here is that of cancellation. Since the scheduling is out of your hands, it is possible that a task may be started after you time out waiting for it, but before cancel() has a chance to do its thing. The result won't get propagated, but if the task has meaningful side effects, this approach could create issues.