Out of Memory Error when using Executors in java

Question

I am using the Executors framework(fixed thread pool with unbounded blocking queue) to execute tasks concurrently.

But when I run a load test with about 10000 tasks created, there is a huge build up of heap memory (2.1 GB) with about 3.5 million Executable objects.

I am not sure if the unbounded queue is causing this build up.

Memory Analyzer report :

One instance of "java.util.concurrent.ThreadPoolExecutor" loaded by "" occupies 2,299,506,584 (94.97%) bytes. The instance is referenced by com.test.ScheduleBean @ 0x743592b28 , loaded by "org.jboss.modules.ModuleClassLoader @ 0x741b4cc40".

Any pointers appreciated!

        //The Executors are loaded in a hashmap
        HashMap<String,Executor> poolExecutorMap = new HashMap<String,Executor>();

       //Executor is a fixed thread pool one
       Executor poolExecutor = Executors.newFixedThreadPool(threadCount);

      //then add the executor to the hashmap
       poolExecutorMap.put("Executor", poolExecutor);



    //then a list of tasks are pulled from a database and passed as runnable objects to the executors

      Class<?> monitorClass=null;

      List<Task> list = getAllTasksToProcess();

     for (int i = 0; i < list.size(); i++) {
      Task task = list.get((int) i);

     monitorClass = Class.forName(task.getTask_event_name());
        Constructor<?> ctor;
        ctor = monitorClass.getConstructor(Task.class);
        Object object = ctor.newInstance(task);
        logger.debug("Adding  task number : "+task.getTask_sequence_id());
        poolExecutorMap.get("Executor").execute((Runnable) object);
}



// the executor classes have an execute method which sends a http notification.

I don't understand how the map is used. There's appears to be only one `Executor` in it, with the key "Executor". But then you are `get()`-ing others, keyed by a class name. Where do these come from? — erickson, Apr 29 '15 at 19:31
I have modified it a bit, assume there is only one executor with the name "Executor" — DntFrgtDSemiCln, Apr 29 '15 at 19:36
Take a look at the retained heap of the Task objects in MemoryAnalyzer. You might want to dig down into the object graph if the number looks high to see where the memory is used. After all, you are creating thousands of them. — Mike Tunnicliffe, Apr 29 '15 at 20:08

score 2 · Answer 1 · answered Apr 30 '15 at 10:59

Write OQL in MemoryAnalyzerTool

select * from java.util.concurrent.ThreadPoolExecutor

and execute the query. It will list the object in the separate window. Then right click on the instances generated then

path to GC roots --> Exclude soft/weak/phantom references

It will help you in understanding who is holding strong reference of the suspected objects.

score 0 · Accepted Answer · answered Apr 30 '15 at 15:02

0

Thanks for all the responses.

The real problem was the way tasks were being pulled from the database. Duplicate tasks were being added to the task queue and hence the queue build up.

answered Apr 30 '15 at 15:02

DntFrgtDSemiCln

1,259
2
16
35

Out of Memory Error when using Executors in java

2 Answers2