2

In a program that tasks are taken from a one file and are executed in parallel i want my program to survive failures / restarts etc. and to resume its execution instead of starting from the beginning each time in case i have a very big list of tasks in the file. What is the proper way of achieving this goal i didn't find much in google to start from. My tasks representation in java are simple pojos with 3 "File" members. Here is my code so far. Thank you for any help in advance.

import java.io.*;
import java.nio.file.*;
import java.util.*;
import java.util.concurrent.*;
import java.util.concurrent.atomic.AtomicInteger;

public class MyTaskManager {
    private static AtomicInteger id;
    private static String[] in;
    private static String cmd;
    private static Task task;
    private static Process process;
    private static MyTaskManager taskManager;
    private static Semaphore sem;
    private static Map<String, Task> taskMap;
    private static StringBuffer sb;
    private static List<Runnable> threadList;


    public static void main(String[] args) throws IOException {
        id = new AtomicInteger();
        taskManager = new MyTaskManager();
        threadList = new ArrayList<>();
        taskMap = new ConcurrentHashMap<>();
        sem = new Semaphore(1);
        for (int i = 0; i < 2; i++) {
            Thread t = new Thread(new Runnable() {
                @Override
                public void run() {
                    try {
                        // Get task
                        synchronized (taskManager) {
                            String line = Files.readAllLines(Paths.get("commands.txt"))
                                    .get(id.get());
                            id.getAndIncrement();
                            in = line.split(" ");
                            cmd = createCmd(cmd);
                            task = new Task(cmd);
                        }
                        // Persist task for later execution if it has missing dependencies
                        if (!task.checkCondition(task)) {
                            sem.acquire();
                            taskMap.put(task.getFile_dep_1().toString().trim(), task);
                            taskMap.put(task.getFile_dep_2().toString().trim(), task);
                            sem.release();
                            // execute the task and see if persisted task can continue his execution
                        } else if (task.checkCondition(task)) {
                            sem.acquire();
                            process = Runtime.getRuntime()
                                    .exec("cmd /c start cmd.exe /k \"" + task.getCmd() + "\"");
                            process.waitFor();
                            String taskKey = task.getOutput_file().toString().trim();
                            System.out.println(taskKey);
                            if (taskMap.containsKey(taskKey)) {
                                System.out.println("In contains key " +
                                        Thread.currentThread().getName());
                                Task taskTmp = taskMap.get(taskKey);
                                process = Runtime.getRuntime()
                                        .exec("cmd /c start cmd.exe /k \"" + taskTmp.getCmd() + "\"");
                                process.waitFor();
                            }
                            sem.release();
                        }

                    } catch (IOException | InterruptedException e) {
                        Thread.currentThread().interrupt();
                        e.printStackTrace();
                    }
                }
            });
            threadList.add(t);
        }

        BlockingQueue<Runnable> worksQueue = new
                ArrayBlockingQueue<>(10);
        RejectedExecutionHandler rejectionHandler = new
                RejectedExecutionHandlerImpl();
        ThreadPoolExecutor executor = new ThreadPoolExecutor(2, 2, 20,
                TimeUnit.SECONDS, worksQueue, rejectionHandler);
        executor.prestartAllCoreThreads();

        worksQueue.add(new MultiRunnable(threadList));

        executor.shutdown();
    }


    private static String createCmd(String cmd) {
        StringJoiner stringJoiner = new StringJoiner(" ");
        stringJoiner.add(in[0]).add(in[1]).add(in[2])
                .add(in[3]).add(in[4]);
        return stringJoiner.toString();
    }
}

class Task {
    private File file_dep_1;
    private File file_dep_2;
    private File output_file;

    .......
}
  • [JVM Shutdown Hooks](https://docs.oracle.com/javase/8/docs/technotes/guides/lang/hook-design.html) ? – Michał Krzywański Aug 22 '19 at 07:13
  • 1
    What you describe looks to me like application checkpoint. Did you take a look at this question before: https://stackoverflow.com/questions/9566486/java-checkpointing? – Kavitha Karunakaran Aug 22 '19 at 07:21
  • 1
    @Kavitha Karunakaran From the question you posted a link to and from another related to it i have found out that i will need to use javaflow API and Continuation but i am not sure. Thank you. –  Aug 22 '19 at 07:40
  • 1
    Write a way to save the state or keep track of what you've already done, and write code to resume from there. If saving is expensive, only do it at regular intervals and be prepared to lose the work from the last saved interval. Or use middleware that does it for you if that fits your problem; like using a persisted queue / topic (Kafka for example) – Erwin Bolwidt Aug 22 '19 at 07:47
  • @BorisBorovski - glad to know that. By the way, your initial question is going to be really useful for people who want to know how to resume execution of tasks from a place they abruptly stopped. – Kavitha Karunakaran Aug 22 '19 at 07:54

0 Answers0