Java BlockingQueue messages not processed not processed in full

Question

I am trying to process few million records from a text file (i.e. reading the file sequentially using one thread, and trying to process the retrieved lines using multiple threads). A method call after 'queue.take();' is only executing for the number of times equal to initial capacity allocated to BlockingQueue (100 in this example), and then the process doesn't pickup anymore records.

Could you please help in debugging or identify the issue?

Main Method:
############
final int threadCount = 10;
BlockingQueue<String> queue = new ArrayBlockingQueue<String>(100);
ExecutorService service = Executors.newFixedThreadPool(threadCount);

for (int i = 0; i < (threadCount - 1); i++) {
    service.submit(new EvaluateLine(queue));
}
service.submit(new ProcessFile(queue)).get();
service.shutdownNow();
service.awaitTermination(365, TimeUnit.DAYS);


EvaluateLine:
#############
private final BlockingQueue<String> queue;

public EvaluateLine(BlockingQueue<String> queue){
    this.queue = queue;
}

@Override
public void run() {
    String line;
    while(true) {
        try {
            line = queue.take();
            SyncOutput.Process(line);
        } catch (InterruptedException ex) {
            break;
        }
    }
}

ProcessFile:
############
private final BlockingQueue<String> queue;

public ProcessFile(BlockingQueue<String> queue) {
    this.queue = queue;
}

@Override
public void run() {
    Path path = Paths.get("C:\\Desktop\\testdata.txt");
    BufferedReader br = null;

    try {
        br =Files.newBufferedReader(path, StandardCharsets.UTF_8);
        for (String line; (line = br.readLine()) != null; ) {
            queue.put(line);
        }
    } catch(IOException e){
        e.printStackTrace();
    } catch(InterruptedException e){
        e.printStackTrace();
    } finally{
        try {
            if (br != null) br.close();
        }catch(IOException e){
            e.printStackTrace();
        }
    }

Edit:

 SyncOutput
 ##########
 class SyncOutput{
    public static void ProcessExpression(String inputLine) {
        evalExpression(inputLine);
    }
    public static double evalExpression(String s){
        Expression e = new ExpressionBuilder(s)
                       .build();
        return e.evaluate();
    }
 }

There are lots of errors in syntax of the `SyncOutput` class (synchronize not synchronized; void method returning value; `SyncOutput.Process` being called statically). Also, the `SyncOutput` class isn't complete, so this code can't be compiled to replicate the issue. If you just replace the `SyncOutput.Process` with `System.out.println` (and remove the `SyncOutput` class), does the same problem still occur? — Andy Turner, Mar 20 '16 at 21:32
@AndyTurner I have removed the additional code and only retained the portion of calling the exp4j api method to evaluate expression. This block appears to be causing the issue. I have replaced the SyncOutput.Process with System.out.println as per your suggestion, and it executes as expected. — rslack, Mar 20 '16 at 21:53
@AndyTurner I have changed the library to parsii and now the processing works as expected. Thanks for your prompt feedback — rslack, Mar 20 '16 at 22:07

score 0 · Answer 1 · answered Mar 21 '16 at 03:49

The following code segment(using Expression evaluation library: exp4j) is blocking the multi-thread execution, not sure why. But I have replaced this code block with a different library(parsii) and now everything looks fine.

public static void ProcessExpression(String inputLine) {
    evalExpression(inputLine);
}
public static double evalExpression(String s){
    Expression e = new ExpressionBuilder(s)
                   .build();
    return e.evaluate();
}

Java BlockingQueue messages not processed not processed in full

Edit:

1 Answers1