0

I have a map of HashMap where Node is a class containing some info and most importantly containing a neighbors HashMap.

My algorithm does random gets from the outer HashMap and puts in the inner HashMap, e.g. 

Node node = map.get(someInt); 
node.getNeighbors().put(someInt, someOtherInt);

I have lots of entries in both the outer map and inner maps. The full dataset cannot be handled by memory alone, so I need to use some disk caching. I would like to use memory as much as possible (until almost full, or until it reaches a threshold I specify) and then evict entries to disk.

I have used several caches (mostly mapDb and EhCache) trying to solve my problem but with no luck. I am setting a maximum memory size, but the cache just ignores it. I am almost certain that the problem lies to the fact that my object is of dynamic size.

Anyone got any ideas on how I could handle this problem? Thanks in advance.

Nick Pavlakis
  • 223
  • 2
  • 8
  • Could give me idea why EhCache could not help you? Does it evict data to disk cache from memory? – sphinks Nov 19 '15 at 09:12
  • EhCache eviction seems to ignore the size limits. I am running with -Xmx1g and setting the memory cache size to 300mb, yet after some time the memory gets filled up and there is too much GC activity, crippling my app. – Nick Pavlakis Nov 20 '15 at 11:41

1 Answers1

0

Note: I work on Ehcache

Ehcache, as most other caching products, cannot know about your object size growing unless you let it know by updating the mapping from the outside:

Node node = cache.get(someInt); 
node.getNeighbors().put(someInt, someOtherInt);
cache.put(someInt, node);

In that context, Ehcache will properly track the object growth and trigger memory eviction.

Note that Ehcache no longer uses an overflow model between the heap and disk but instead always stores the mapping on disk while keeping a hotset on heap.

Louis Jacomet
  • 13,661
  • 2
  • 34
  • 43
  • Thanks for your reply. I have done that too. Still no luck. I am running with -Xmx1g and setting the memory cache size to 300mb, yet after some time the memory gets filled up and there is too much GC activity, crippling my app. I am using EhCache 2.9 – Nick Pavlakis Nov 20 '15 at 11:40
  • Did you take a memory dump? Is Ehcache responsible for the GC activity or is the pressure coming from your app? – Louis Jacomet Nov 20 '15 at 17:35
  • Yes, I took a heap dump through VisualVM. Most objects are byte[] and char[] and I don't use any of those in my app. I am guessing that those are the serialized versions of my objects. – Nick Pavlakis Nov 21 '15 at 19:56