2

My application correctly runs correctly on a server with 16 processor and 64 GB of Ram. I have multiple process and I try to limit the maximum heap at 8 Gbyte for my process.

My problem is that I have some form of producer-consumer pattern and I have to limit the producing rate or I will run out of memory, because the garbage collection for old generation occurs too rarely.

  • When I monitor my application, I can see that after being running for 4 hours I have spent 7 minutes in ParNEW and 0.775 seconds in ConcurrentMarkSweep.
  • My heap occupation gets up to around 6GB then drops to 1GB then slowly start going up to 6GB then drop down again. The cycles are approximately of 10 minutes

Through JVisualVM I can see that 90% of my memory occupation is given by CMS Old Gen, how can I force the concurrent Mark sweep to run a little bit more frequently?

@PeterLawrey comment is very relevant, since my application runs on the top of an Application server which is designed for event-driven processing and data partitioning, such as Terracotta or Coherence. It is likely that the underlying implementation includes a queueing system for event-processing.

My problem is that limiting the heap size is not a solution, because as I had experienced, instead of having more frequent garbage collection, the application runs out of memory.

Edmondo
  • 19,559
  • 13
  • 62
  • 115
  • 2
    A Full GC is always run before you get an OOME so it doesn't matter how often CMS runs. Is this what you mean? or do you mean something else by "out of memory" – Peter Lawrey Sep 17 '12 at 14:54
  • 2
    why do you think it needs to collect more frequently? why do you think it will run out of memory? what behaviour do you class as being OoM? – Matt Sep 17 '12 at 14:54
  • I am not entirely sure if imposing Producer's production limit by heap size is the best choice possible. Such a parameter should be explicitly specified to reflect some absolute maximum value of created objects. EDIT: See @Peter Lawrey answer – Wojciech Owczarczyk Sep 17 '12 at 15:00
  • Needing the old generation to be collected often sounds like a design smell to me. Objects with temporary lifetimes should rarely make it there to begin with. – millimoose Sep 17 '12 at 15:03
  • If you limit your heap memory to, say, 2GB, then it would run gc every 2 minutes. Each gc call would consume less CPU time but total CPU time would be more than 7 min/4h. But will you or will not run out of memory does not depend on how often gc is called - it depends solely on the real consumption of memory which is 1 GB in your case. – Alexei Kaigorodov Sep 17 '12 at 16:38
  • I added some details as @PeterLawrey introduces a very relevant point – Edmondo Sep 18 '12 at 06:55
  • Using a memory profiler or even `jmap -histo:live` may be enough to see that a queue is filling up the memory. – Peter Lawrey Sep 18 '12 at 08:50
  • Yes I see that and I have to reduce my producing rate. How could I instead decide to cleanup the space through the GC more often? – Edmondo Sep 18 '12 at 09:32

2 Answers2

3

I have to limit the producing rate or I will run out of memory

This would happen if you use unbounded queues. If your producer exceeds the rate of the consumer, the queue can grow without limit until you run out of memory. Limiting the producer can ensure the queues stay at a reasonable size.

A simple way around this is to delay the producer when the queue gets too long.

private final BlockingQueue<Task> tasks = new ArrayBlockingQueue<Task>(1000);

// the producer slows when the queue length gets too long.
tasks.offer(newTask, 1, TimeUnit.HOURS);

This will ensure the queue never gets too long but without slowing the producer unnecessarily.

BTW: Using an ArrayBlockingQueue can also reduce the amount of garbage produced compared with Linked Queues.

If you really need an unbounded queue, you can use a library I wrote but it relatively low level. Java Chronicle The producer can be more than the size of the main memory ahead of the consumer. Its only bounded by the size of your disk space.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
2

I think by setting lower initiating occupancy you can trigger frequent collections. 

-XX:CMSInitiatingOccupancyFraction=<nn>
kosa
  • 65,990
  • 13
  • 130
  • 167