Thread to process multiple rest calls

Question

I am trying to process around 1000 files using below code:

ExecutorService executor = Executors.newFixedThreadPool(threadPoolSize);
Runnable worker = null;
for (File file : files) {
    if (file.isFile()) {
        worker = new FileProcessThread(file, connectionVo);
        executor.execute(worker);
        file.deleteOnExit();
    }
}
while (!executor.isTerminated()) {
    System.out.println("Still running");
}
executor.shutdown();
System.out.println("Finished all threads");

This code creates multiple threads. Each thread has multiple rest calls inside. These rest apis are for processing input file. Each thread also logs each transaction event which occurs while processing. But result of these threads execution is not consistent.

For few threads it works perfectly fine.Picks the file.Logs correct
transactions and move processed file to proper directory.
But for for some threads it shows some unpredictable behavior such as it logs file process event of one thread into other.

Steps in each thread :

Create transaction - rest call
Log event in transaction for process start - rest call
Gives file to other module for file conversion - rest call which internally
creates one more thread which is synchronized
Once file is processed it is moved to other - in the same code directory

I want consistent performance out of these threads. Any help will be appreciated.

Code inside run :

 long transactionID = 0l;
    long connectionId = connectionVo.getConnectionId();
    try {
        transactionID = beginTransaction.getTransactionId();
        FileInputStream processedFileData;
        processedFileData = new FileInputStream(file);

        Response response = Service.postMessage(stream2file,              
        connectionId, 0, transactionID);

        if (response.getStatus() != 200) {
              writToDirectory(stream2file, userError, file.getName(), transactionID);
            }

        } else {
            String userArchive = getUserArchive();
            if (checkDirectory(userArchive, transactionID)) {
                writToDirectory(stream2file, userArchive, file.getName(), transactionID);
            }

        }
        file.delete();

    } catch (FileNotFoundException e) {
    }

I think your `threadPoolSize` must be less than 1000 which is the amount of the files to be processed. So thread will be reused to deal with different files. — tianwei, Jan 12 '16 at 09:25
Also, if I get it - its only about the log which is getting messed up. If this is the case, then you can look into `Future`. Submit your task to the executor service to get a future reference back and using it query the result. You can try returning a completion string which in your case could be `rest call name - log event - tran id` and log it. — Tirath, Jan 12 '16 at 09:44
you could also add the thread id and the name of the file being processed to the logged message that would make it clearer what message comes from which thread — Gavriel, Jan 12 '16 at 09:58
You might also consider posting the code for FileProcessThread, as that is probably what's causing the problem. You probably use some non-thread safe calls in there. — Gavriel, Jan 12 '16 at 10:02
The code you show here doesn't tell anything about the problem. It has to do with how `FileProcessThread` is implemented. You are passing `connectionVo` which sounds suspicious (generally you don't share connections across threads) — Enno Shioji, Jan 12 '16 at 10:07
You delete files on exit, not when successfully processed so if your application dies before each file is processed they can get deleted anyway. I suggest you only delete files which were successfully processed and you move files which fail into another directory. — Peter Lawrey, Jan 12 '16 at 11:31

score 1 · Answer 1 · answered Jan 12 '16 at 11:34

I suggest you use Java 8 to do the multi-threading as it is much cleaner.

files.parallelStream()
     .filter(File::isFile)
     .forEach(f -> new FileProcessThread(file, connectionVo).run());

Your task deletes the file when finished successfully.

This will only pass each file to one task.

BTW Don't call your tasks xxxThread unless they are actually a Thread, and avoid ever sub-classing a Thread.

Thread to process multiple rest calls

1 Answers1