Caffeine read/write taking longer under high load

Question

This is how I build my Caffeine Cache in Spring boot:

Caffeine.newBuilder()
                .initialCapacity(200000)
                .maximumSize(200000)
                .expireAfterWrite(15, TimeUnit.MINUTES)
                .scheduler(Scheduler.systemScheduler())
                .executor(caffeineWriteClientConnectionPool(cacheSpec))
                .recordStats(() -> caffeineMetrics)
                .writer(new CacheWriter<Object, Object>() {});

I use the following Synchronous methods:

cache.getAllPresent() -> fetch all keys present in cache if not fallback to distributed cache/db and then add the key to the caffeine cache
cache.putAll() -> a key is added to the caffeine cache when its missed

When the cache load is high, I see get/put time increasing(>100ms) and this is making me think if these calls should be behind hystrix but I am think that's gonna be an overkill spinning up a thread for an in-memory search?

Also, I see the GC happening pretty often and I believe this might be impacting the read/write time. Any suggestions/best practices on setting up the GC or any other general suggestion I can do to improve the latency? thanks

To reduce locking and latency, use `getAll` with a loader instead of a racy read and explicit put. This avoids redundant work (cache stampede) and reduces locking. An async cache can better optimize bulk loads, but generally at only a slight improvement. — Ben Manes, Apr 12 '20 at 20:37
getAll/putAll might call anywhere between 1-100 keys at a time under high load. I see the ```getAll``` method under the hood calls ```getAllPresent``` and then loads the keys that are not present. If a key is absent in Caffeine, the logic falls back to the distributed cache/db where I explicitly add the missing keys using ```putAll``` to Caffeine. After considering these, do you still think that using ```getAll``` would reduce latency/locking or suggest using Async? — Avinash Reddy Penugonda, Apr 12 '20 at 21:17
On that's right. In async, it is able to be smarter by inserting proxy futures that are filled when a single bulk call is made. This way only bulk call is made and subsequent calls obtain an in-flight future for those keys. Unfortunately we can't do similar in a sync cache for `getAll`, but you can use `AsyncCache.synchronous()` to have a sync view. That might help, e.g. [doordash](https://doordash.engineering/2018/08/03/avoiding-cache-stampede-at-doordash/) reported an 87% latency reduction by avoiding stampedes. — Ben Manes, Apr 12 '20 at 21:23
I'll try implementing the ```getAll``` with a loader using the ```AsyncCache.synchronous()``` view and see if it helps improve the performance. Thanks! — Avinash Reddy Penugonda, Apr 13 '20 at 15:43
@BenManes While looking at the loader for ```getAll``` in ```AsyncCache.synchronous()``` view, the loader function accepts the keys as an argument and return the values. But the L2 cache/db as the fallback methods require more info/arguments such as the logger which is initialized with the data from the client request and some other flags, do you think this is achievable using the Loader function? It seems I have the implement the locking/proxy futures? any suggestions please? — Avinash Reddy Penugonda, Apr 13 '20 at 22:49
The lamda can capture the contextual state, so you can pass it through there. The cache only stores they `key` => `future` mapping, but the lambda function can capture the outer scope, e.g. `var userId = 123; cache.getAll(dataKeys, keys -> load(keys, userId))` — Ben Manes, Apr 13 '20 at 23:45
@BenManes the first test on the ```getAll``` method in the ```AsyncCache.synchronous()``` view seems to perform a lot better, thanks for the help — Avinash Reddy Penugonda, Apr 15 '20 at 01:06
@BenManes Appreciate your efforts in answering all the questions, using AsyncCache resolved the latency issues we had before but one question I have though is using the AsyncCache using a thread pool seems to be an overkill as we are already calling the cache from a different thread pool thread, so do you think using the Cache ```getAll``` with a loader would give almost the same performance without the additional thread overhead? I am worried that we might see cache stampede with using Cache with loader? Wanted to know your thoughts before I test. Thanks — Avinash Reddy Penugonda, Apr 22 '20 at 21:17
You can specify the threadpool used by the cache or return a future from the loader that it will store. You could try using a same thread executor (Runnable::run) which would probably give you the results you want. — Ben Manes, Apr 23 '20 at 00:55
I was able to do it and everything's working as expected, thanks a lot for the suggestions. you're awesome! — Avinash Reddy Penugonda, May 06 '20 at 01:33

Caffeine read/write taking longer under high load

0 Answers0