Caffeine AsyncLoadingCache and thundering herd

Question

Does .get() on a Caffeine AsyncLoadingCache prevent concurrent loads, by delaying subsequent threads which are calling .get() until the first one completes? Or that it can be configured to return a stale value while a self-populating load request is occurring?

This is so that a thundering herd can be prevented by using the cache.

I am seeing behavior which indicates that the thundering herd is not handled even though I am using a cache.

I create the cache like so:

val queryResponseCache: AsyncLoadingCache<Request, Response> = Caffeine.newBuilder()
            .maximumSize(1000)
            .expireAfterWrite(5, TimeUnit.SECONDS)
            .recordStats()
            .buildAsync(queryLoader)

And use it in conjunction with a L2 cache in redis like so (kotlin elvis operator):

queryResponseCache.getIfPresent(key) ?: fetchFromRedis(key) ?: queryResponseCache.get(key)

I understand that getIfPresent is concurrent, but the subsequent calls which end up calling fetchFromRedis() / get() seem to have problems. I guess moving the fetchFromRedis into the asyncLoad() function might be better for load tolerance.

score 2 · Accepted Answer · answered Mar 18 '20 at 21:13

A cache stampede is supported when you load through the cache. In your example using getIfPresent and loading the value, then I assume you put it into the cache explicitly inside of fetchFromRedis. Either way, you are ensuring a racy get-load-put due to bypassing the cache except when absent in Redis.

If you move the logic into asyncLoad, as you surmised, it would let the cache handle the stampede. The redis lookup, db query, and storing back into redis can all be performed as a chain of asynchronous tasks where the final future is returned to asyncLoad. Then the cache will compute the future once and return it to all subsequent calls until the entry is evicted.

Caffeine AsyncLoadingCache and thundering herd

1 Answers1