Why Java BlockingQueue virtually stops growing up in my program?

Question

I have a file processing program written in Java. The processed data will be put into a LinkedBlockingQueue for their consumers. While I debugged, I stopped all consumers just to make it simple. I noticed that almost every time when the size of that blocking queue reached around 3600ish, everything became slower and slower and then just virtually stopped (the program didn't exit, but no queue size output anymore, and no file got processed) without any OutOfMemoryException. And from the resource monitor, I saw memory level stayed very stable.

Does anybody know what the possible causes could be?

//Here's the code: In Main:

_taskQueue = new BlockingTaskDQueue<StreamQueueItem>();
_fileProcessor = new LogFileProcessor(_taskQueue);
(new Thread(_fileProcessor,LogFileProcessor.myName)).start();

LogFileProcessor:

public class LogFileProcessor implements Runnable {
public static final String myName = ConfigConstants.THREAD_NAME_PREFIX + "FileProcessor";

private BlockingTaskDQueue<StreamQueueItem> mTaskQueue;

private ExecutorService mThreadPool;

//indicate whether the thread pool has been shut down
private boolean mIsEnded = false;

public LogFileProcessor(BlockingTaskDQueue<StreamQueueItem> _taskQueue) {
    super();
    this.mTaskQueue = _taskQueue;


    ThreadFactory namedThreadFactory = new ThreadFactoryBuilder().setNameFormat(ConfigConstants.THREAD_NAME_PREFIX + "FileProc-%d").build();
    this.mThreadPool =Executors.newSingleThreadExecutor(namedThreadFactory);
}

@Override
public void run() {

    File[] files = null;

    //load files in the live_folder
    files = ConfigConstants.LIVE_FOLDER.listFiles();
    if(files!=null){
        for(File f:files){
            FileProcTask task = new FileProcTask(f.getName());
            mThreadPool.submit(task);
        }
    }

    // Filter which does not accept .tmp files
    FileExtensionFilter filter = new FileExtensionFilter("tmp");
    while (true) {
        // get the non-temp txt file list
        files = ConfigConstants.PRODUCTION_FOLDER.listFiles(filter);

        if (files.length == 0) {
            continue;
        }

        for (File f : files) {
            //move file to the live folder first, then start the task
            FileOperator.move(f, ConfigConstants.LIVE_FOLDER);
            FileProcTask task = new FileProcTask(f.getName());
            mThreadPool.submit(task);
        }


    }

  }

}

Inner FileProcTask class:

    class FileProcTask implements Runnable {
    private File mFile;
    private String mCurrentFileName;
    private HashMap<LogReason, List<String[]>> mTableRowMap;

    public FileProcTask(String fileName) {
        mCurrentFileName = fileName;
        mFile = new File(ConfigConstants.LIVE_FOLDER.getAbsolutePath()+ File.separator+fileName);
        this.mTableRowMap = new HashMap<LogReason, List<String[]>>();
    }

    @Override
    public void run() {     
        boolean success = true;         
        FileInputStream fis;
        BufferedReader reader=null;
        try {
            fis = new FileInputStream(mFile);
            reader = new BufferedReader(new InputStreamReader(fis));
            String recordItem = reader.readLine(); // ignore the first line of log file
            LogReason logReason= LogReason.Unknown;
            while(recordItem!=null){
                recordItem = reader.readLine();
                ConfigConstants.IncRowsScanned();
                logReason = this.getLogReason(recordItem);
                if(logReason != LogReason.Unknown && logReason != null){
                     insertToQueue(mCurrentFileName,logReason
                                    ,this.getColumns(recordItem));
                }
            }
        } catch (FileNotFoundException e1) {
            success = false;
            e1.printStackTrace();
        } catch (IOException e) {
            success = false;
            e.printStackTrace();
        }finally{
            if (reader!=null)
                try {
                    reader.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
        }
        if(success){
            // move file to archive folder
            String moveToFolder = FileOperator.getArchiveFolder(mFile);
            FileOperator.move(mFile, new File(moveToFolder));

            //flush all the rest rows into stream task queue
            List<String[]> list_tmp = null;
            for (LogReason lr_tmp : mTableRowMap.keySet()) {
                list_tmp = mTableRowMap.get(lr_tmp);
                if (list_tmp != null && list_tmp.size() > 0) {
                    synchronized(mTaskQueue){
                         try {
                            mTaskQueue.putNewTask(new StreamQueueItem(lr_tmp,
                                                   new ArrayList<String[]>(list_tmp)));
                            list_tmp = null;
                            System.out.println(mTaskQueue.size());
                        } catch (InterruptedException e) {
                            e.printStackTrace();
                        }
                    }
                }
            }

        //long now = System.nanoTime();
        //ConfigConstants.addToInterval(now-start);
        //System.out.println(mCurrentFileName + " process done##");
        }else{
            System.out.println(mCurrentFileName + " move back##");
            FileOperator.move(mFile, ConfigConstants.PRODUCTION_FOLDER);
            System.out.format("$$ File %s processing failed, move it back to production folder for re-try",mCurrentFileName);
        }
    }

    private void insertToQueue(String currentFileName, LogReason logreason, String[] columnValues) {
        if(columnValues==null)
            return;
        List<String[]> currentList = mTableRowMap.get(logreason);
        if(currentList==null){
            currentList = new ArrayList<String[]>();
            mTableRowMap.put(logreason, currentList);
        }
        if (currentList.size() >= ConfigConstants.ROWS_PER_REQUEST) {
            // when the current list reach the request quota, put the Bigquery
            // Row list into the blocked stream queue.
            synchronized(mTaskQueue){
                try {
                    mTaskQueue.putNewTask(new StreamQueueItem(logreason, new ArrayList<String[]>(currentList)));
                    currentList.clear();
                } catch (InterruptedException e) {
                    e.printStackTrace();
                    return;
                }
            }
        } 
        currentList.add(columnValues);
    }
}

In the last class, there is a statement of "System.out.println(mTaskQueue.size());". It's where I tried to output the queue size.

It's probably caused by some problem in the code. Why don't you share it: it would allow us to find the problem, instead of guessing what it could be. — JB Nizet, Jul 07 '14 at 19:42
See the BlockingQueue documentation, particularly if you're using put() to add to it: "A BlockingQueue may be capacity bounded. At any given time it may have a remainingCapacity beyond which no additional elements can be put without blocking." You are likely reaching a capacity that results in blocking. If you try offer() or add() instead you'll get an exception if it fails (and that will answer your question definitively). — user1676075, Jul 07 '14 at 20:02
@user1676075 Taken your advise, I tried to use offer and print out the result. Still, when I saw "virtually stop", it went with true, but just gradually getting slower and no print out...(at least within a veeeeery long period of time) — foxwendy, Jul 07 '14 at 20:14
LinedBlockingQueue should not be a capacity bounded, isn't it? (Unless it reaches Integer.Max_Value or get the JVM out of memory.) — foxwendy, Jul 07 '14 at 20:16
Given that you have a single thread executor, and that the problem always occurs at the same number of tasks, my guess is that it's not a queue or multi-threading problem, but that it's a file descriptor problem: you probably forget to close a stream somewhere, which causes the number of open file descriptors until the limit is reached. — JB Nizet, Jul 07 '14 at 20:29
Your timeout matters (did you wait beyond the timeout time?). Also try put(), because that will throw an exception immediately on error (although a timeout of zero would effectively be the same). Are you running up against memory limits? Do you definitely not have any catch blocks that are handling (and hiding) memory errors? — user1676075, Jul 08 '14 at 15:40
Only thing that looks like an error in the code is this: "synchronized(mTaskQueue){". If you're using a BlockingQueue, you shouldn't separately sync on it; that will all be handled internally. Although that should result in stoppage, not slowdown, if it's causing problems. — user1676075, Jul 08 '14 at 15:46
This morning, I ran this program again, after a while, it gave me "Exception in thread "Thread-FileProcessor" java.lang.OutOfMemoryError: GC overhead limit exceeded". What could it possibly imply? Memeory leaking? I used if(reader!=null) reader.close when the reaches the file end. How can I locate which statement is leaking memory, any tools can helped on that? — foxwendy, Jul 08 '14 at 17:51
It is probably the put() into the blocking queue that's failing. When the blocking queue allocates memory, it needs to allocate a new block, then free the existing one. To make room the GC needs to free and move around other items. That's too busy, can't be done in enough time, you get that error. If your base is ArrayBlockingQueue, try allocating a max size by default (then allowing the add to block until space is available), or consider LinkedBlockingQueue which will allocate space in smaller chunks. — user1676075, Jul 08 '14 at 22:04
You can always use a profiler to inspect the heap, which would give you some idea of what's taking up the most memory. — awksp, Jul 10 '14 at 03:25

score 0 · Answer 1 · answered Jul 11 '14 at 20:55

Thanks for the suggestion. I ended up cross-refer to JVM memory model. When the BlockingQueue tries to put more items in before it reached its capacity, it's competing with other threads (either application threads or system native threads invoked by my application) for JVM heap space. So, program will stuck at somewhere when this competition happens and crash when GC is not able to find any available heap space with OutofMemory exception. And,this exception will happen after quite a while the program got stucked. I assume that's why I didn't see any exception at queue put/offer.

Why Java BlockingQueue virtually stops growing up in my program?

1 Answers1