Java. Consumer - Producer with BlockingQueue. Search tool

Question

I was trying to implement some Consumer-Producer problem with BlockingQueue. To do it with some purpose, I decided to write file searching tool.

I decided that search mechanism is working recursively, and every new directory is going to have new thread pool to increase speed of searching.

My problem is, that I have no idea how can I implement mechanism that stops printing threads (consumers) at the end - when searching threads get job done.

I was trying to do that with some ideas like POISON PILLS, but it doesn't works well (threads stops before print any results). Any ideas how can I do that?

Here is some code:

Searching mechanism:

public class SearchingAlgorithm implements Runnable {

private final File file;
private BlockingQueue<File> queue;
private ExecutorService executor;

public SearchingAlgorithm(File fileName, BlockingQueue<File> queue) {
    this.file = fileName;
    this.queue = queue;
    this.executor = Executors.newWorkStealingPool();
}

@Override
public void run() {
    try {
        searchDeep();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

private void searchDeep() throws InterruptedException {
    File[] files = file.listFiles();
    if (files != null) {
        for (File fil : files) {
            if (fil.isDirectory()) {
                executor.submit(new SearchingAlgorithm(fil, this.queue));
            } else {
                this.queue.add(fil);
            }
        }
    }
}

}

Printer:

public class ContainingCheckAlgorithm implements Runnable {

private BlockingQueue<File> queue;
// private ExecutorService executor;
private String keyWord;

public ContainingCheckAlgorithm(BlockingQueue<File> queue, String keyWord) {
    this.queue = queue;
    this.keyWord = keyWord;
    // executor = Executors.newFixedThreadPool(2);
}

@Override
public void run() {
    try {
        printFile();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

}

private void printFile() throws InterruptedException {
    while (true) {
        File takeFile = queue.take();
        String fileName = takeFile.getAbsolutePath()
                .toLowerCase();
        boolean isContainingKeyWord = fileName.contains(keyWord.toLowerCase());

        if (isContainingKeyWord) {
            System.out.println(takeFile.getAbsolutePath());
        }
    }
}

}

Main test class:

public class MainClass {

public static void main(String[] args) throws InterruptedException {
    ExecutorService executor = Executors.newFixedThreadPool(2);
    BlockingQueue<File> queue = new LinkedBlockingQueue<>();

    File fileName = new File("C:/");

    SearchingAlgorithm sa = new SearchingAlgorithm(fileName, queue);
    executor.submit(sa);

    ContainingCheckAlgorithm ca = new ContainingCheckAlgorithm(queue, "Slipknot");
    executor.submit(ca);

    executor.shutdown();
}

}

score 0 · Answer 1 · answered Jun 07 '17 at 21:35

Split the whole work in 2 stages. At the first stage, SearchingAlgorithm's work and ContainingCheckAlgorithm waits for new jobs if the queue is empty. At second stage, all SearchingAlgorithm instances finished, and ContainingCheckAlgorithm quits if finds the queue empty. To discover if the queue is empty, ContainingCheckAlgorithm uses queue.poll(timeout) instead of queue.take().

And you need not to create new thread pool for each SearchingAlgorithm.

score 0 · Answer 2 · answered Jun 08 '17 at 14:41

As u said, I try to do it this way:

Searching algorithm share Thread Pool with others insatances of searchingAlgorithm.

SEARCHING:

public class SearchingAlgorithm implements Runnable {

private final File file;
private BlockingQueue<File> queue;
private ExecutorService executor;

public SearchingAlgorithm(File fileName, BlockingQueue<File> queue, ExecutorService executor) {
    this.file = fileName;
    this.queue = queue;
    this.executor = executor;
}

@Override
public void run() {
    try {
        searchDeep();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

private void searchDeep() throws InterruptedException {
    File[] files = file.listFiles();
    if (files != null) {
        for (File fil : files) {
            if (fil.isDirectory()) {
                executor.submit(new SearchingAlgorithm(fil, this.queue, executor));
            } else {
                this.queue.add(fil);
            }
        }
    }
}

Now ContainingCheckAlgorith needs to share CountDownLatch with main class, because I need some mechanism to close Thread Pool in main class. Also it uses pool(timeout) as u said, and my threads finally finishing thier job.

CHECKING

public class ContainingCheckAlgorithm implements Runnable {

private BlockingQueue<File> queue;
private String keyWord;
private CountDownLatch latch;

public ContainingCheckAlgorithm(BlockingQueue<File> queue, String keyWord, CountDownLatch latch) {
    this.queue = queue;
    this.keyWord = keyWord;
    this.latch = latch;
}

@Override
public void run() {
    try {
        printFile();
        latch.countDown();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }

}

private void printFile() throws InterruptedException {
    File takeFile;
    while ((takeFile = queue.poll(1, TimeUnit.SECONDS)) != null) {
        String fileName = takeFile.getName()
                .toLowerCase();
        boolean isContainingKeyWord = fileName.contains(keyWord.toLowerCase());

        if (isContainingKeyWord) {
            System.out.println(takeFile.getAbsolutePath());
        }
    }
}

MAIN:

public class MainClass {

public static void main(String[] args) throws InterruptedException {
    ExecutorService executor = Executors.newCachedThreadPool();
    BlockingQueue<File> queue = new LinkedBlockingQueue<>();
    CountDownLatch latch = new CountDownLatch(1);

    File fileName = new File("C:/");

    SearchingAlgorithm sa = new SearchingAlgorithm(fileName, queue, executor);
    executor.submit(sa);

    ContainingCheckAlgorithm ca = new ContainingCheckAlgorithm(queue, "Slipknot", latch);
    executor.submit(ca);

    latch.await();
    executor.shutdown();
}

It looks weird, but I wonder what if:

More than 1 thread would run as ContainingCheckAlgorithm ?
SearchingAlgorithm would searching for an file more than 1 second, and ContainingCheckAlgorithm finish work ? Obviously, I can change timeout to 2 second, and more, but we always try to optimize our programs.

Java. Consumer - Producer with BlockingQueue. Search tool

2 Answers2